You do not need to use a class Edge. You can use adjacency lists and still present an unweighted graph . For a weighted graph, you need a way to represent the value of the edge, and therefore using a class Edgewould be appropriate.
class Graph<E> {
private List<Vertex<E>> vertices;
private static class Vertex<E> {
E elem;
List<Vertex<E>> neighbors;
}
}