Work is a process by which energy changes forms.
What this means, more or less, is that work is not a "thing" or an object, and it isn't really "something that you do to something". It is a process by which something happens.
When work is done, energy changes forms. For example, when you raise your physics book vertically above your desk, we say that "you do work on it". This means, really, that energy is being converted from chemical potential energy stored in your muscles to gravitational potential energy of the book - energy is changing forms. When you let go of the book, gravity "does work on the book". In other words, gravitational potential energy is being converted into kinetic energy - work is changing forms. Work is the process by which energy changes forms.
Conversely, if energy does not change forms, then no work was done. If you carry your book across the room, the energy of the book does not change - no energy changed forms, so no work was done.