A common misconception is that sign languages are like mime or gesture. Signs are in technical jargon “arbitrary”, just like spoken words are. What this means is that words and signs do not necessarily have any logical or obvious or meaningful relationship to the concepts they identify. They might, but that’s not typical.
Words means things because we all agree they mean things, not because any sequence of sounds is inherently meaningful. There is no real reason why cats are called cats – there is nothing to the sequences of phonemes [k a t] that in any way resembles cats or is even reminiscent of cats. And there’s nothing about those individual sounds that is particularly cat-like either. In fact, you can rearrange them into the words “act” and “tack”, and none of those three words have anything to do with each other. Using that word to refer to cats is just an arbitrary convention speakers of English agree on. Any word would do, as is illustrated by the fact that different languages use different words to refer to the same thing, and the names of things can change over time and between social groups.
Similarly, signs have no necessary connection to the things that they refer to. There is nothing inherently meaningful about any particular sequence of hand shapes. And besides, there are are signs for more abstract grammatical concepts like verb agreement, which would be difficult to mime. It has been demonstrated in the lab that monolingual speakers of English perform at chance when trying to identify ASL signs, but if you can experience this arbitrariness yourself: go to YouTube and check out an ASL vlog. Then try to guess what’s going on.
There are of course exceptions to this arbitrariness. For instance, there is onomatopeia, in words like “buzz” or “wham” and in some animal calls (although different languges use different words for animal sounds too, frogs say “kvack” in German, so this too can be arbitrary). Note that the arbitrariness starts to disappear in the semantic domain of “noises”. This strikes me as completely natural. It’s very hard to make phonetic words which resemble physical objects, but it’s easy to make words which sound like sounds.
But in the case of sign languages, you can make words which bear some resemblance to physical objects because it’s a visual language. You can get “visual onomatopeia”, e.g. downward movements for a verb like “to sit” and upward movement for a verb like “to stand up”. And it might even seem, to an observer, like there is a lot of this, but then a lot of our experience with the world is through vision, and languages are crammed full of words that refer to physical objects and physical activities. But which aspect of these objects or activities get represented in a sign is still arbitrary. A nice example of this arbitrariness is this image, which shows the word for “tree” in several different sign languages:
That image is this excellent book, p. 21. I would really encourage anyone interested in sign language to download at least chapters 1, 2 and 3 (reading them after you download would be good too). It gives a fantastic overview of the research that has been done on sign, and while it is a scientific work, it is not a hard read and assumes little to no knowledge on the part of the reader.