Initially, I had the sole theory in my head to the difference between pointers and references: "you can have a pointer to a pointer, but you can't have a reference to a reference".  While that turned out to be true, that was just the beginning of the huge can of worms I uncovered.  But, I'll try to put it as simply as clearly as possible: C++ has pointers and references; Java only has references, but Java references are not the same as C++ references; Java references actually share more similarities with C++ pointers than with C++ references.

First, I found a very good article about pointers vs references in C++ (linked below).  As a definition, pointers and references (in C++) are defined as such:

Pointers: A pointer is a variable that holds memory address of another variable. A pointer needs to be dereferenced with * operator to access the memory location it points to.

References: A reference variable is an alias, that is, another name for an already existing variable. A reference, like a pointer is also implemented by storing the address of an object. A reference can be thought of as a constant pointer (not to be confused with a pointer to a constant value!) with automatic indirection, i.e the compiler will apply the * operator for you.

At this point, Java references do fit the same definition.  However, when we consider the differences between C++ pointers and references, something quite alarming is revealed:

1. Reassignment: A pointer can be re-assigned... On the other hand, a reference cannot be re-assigned, and must be assigned at initialization.
2. Memory Address: A pointer has its own memory address and size on the stack whereas a reference shares the same memory address (with the original variable) but also takes up some space on the stack. References may be passed to functions, stored in classes, etc. in a manner very similar to pointers.  Pointer is an independent variable and can be assigned NEW address values; whereas a reference, once assigned, can never refer to any new object until the variable goes out of scope.
3. NULL value: Pointer can be assigned NULL directly, whereas reference cannot. The constraints associated with references (no NULL, no reassignment) ensure that the underlying operations do not run into exception situation.
4. Indirection: You can have pointers to pointers to pointers offering extra levels of indirection. Whereas references only offer one level of indirection.
5. Arithmetic operations: Various arithmetic operations can be performed on pointers whereas there is no such thing called Reference Arithmetic. 

This was when I got the hunch: Java references are not the same as C++ references, and this was the source off my confusion for all these years.  I sought for affirmation.  It came with the second article linked.  Below are some quotes:

Pass-by-value: The actual parameter (or argument expression) is fully evaluated and the resulting value is copied into a location being used to hold the formal parameter's value during method/function execution. That location is typically a chunk of memory on the runtime stack for the application (which is how Java handles it), but other languages could choose parameter storage differently.

Pass-by-reference: The formal parameter merely acts as an alias for the actual parameter. Anytime the method/function uses the formal parameter (for reading or writing), it is actually using the actual parameter. Java is strictly pass-by-value, exactly as in C. Read the Java Language Specification (JLS). It's spelled out, and it's correct. In http://java.sun.com/docs/books/jls/third_edition/html/classes.html#8.4.1:

When the method or constructor is invoked (15.12), the values of the actual argument expressions initialize newly created parameter variables, each of the declared Type, before execution of the body of the method or constructor. The Identifier that appears in the DeclaratorId may be used as a simple name in the body of the method or constructor to refer to the formal parameter.

In short: Java has pointers and is strictly pass-by-value. There's no funky rules. It's simple, clean, and clear.

The problem here is that the folks at Sun made a naming mistake.  In programming language design, a "pointer" is a variable that indirectly tracks the location of some piece of data. The value of a pointer is often the memory address of the data you're interested in. Some languages allow you to manipulate that address; others do not.  A "reference" is an alias to another variable. Any manipulation done to the reference variable directly changes the original variable.  Check out the second sentence of http://java.sun.com/docs/books/jls/third_edition/html/typesValues.html#4.3.1.

The reference values (often just references) are pointers to these objects, and a special null reference, which refers to no object

They emphasize "pointers" in their description... Interesting...  When they originally were creating Java, they had "pointer" in mind (you can see some remnants of this in things like NullPointerException).  Sun wanted to push Java as a secure language, and one of Java's advantages was that it does not allow pointer arithmetic as C++ does.  They went so far as to try a different name for the concept, formally calling them "references". A big mistake and it's caused even more confusion in the process.  There's a good explanation of reference variables at http://www.cprogramming.com/tutorial/references.html. (C++ specific, but it says the right thing about the concept of a reference variable.)  The word "reference" in programming language design originally comes from how you pass data to subroutines/functions/procedures/methods. A reference parameter is an alias to a variable passed as a parameter.  In the end, Sun made a naming mistake that's caused confusion. Java has pointers, and if you accept that, it makes the way Java behaves make much more sense.

Hence the confusion: because Java references are actually more like C++ pointers.  So, when asked the question what's the difference between pointers vs references, I was having problem differentiating between C++ pointers and Java references.

References:

Pointers vs References in C++

Java is Pass-By-Value