They were afraid, they served most closely with him, but the general public didn't know who he was and if he was some pirate king, he'd have been a famous name, yet he wasn't in TI. Captain Flint was still a feared and famous name. So KatWay is right it doesn't really make sense. What might make sense is this big built up and then...it ends...there is NO pirate king of Nassau or otherwise in the Caribbean, this is the beginning of the end for the pirates really. All that's left is "what almost was" for Silver in terms of that. The build up being more for the sake of irony.
Billy is afraid probably because he's also a crazy old drunk by then and remembers Silver crushing Dufresne's head and saying he has "a long ...... memory". And now he has something against Billy. Flint, I think if the speculation about Thomas is at all right, it would be less a physical fear and more because Silver, sometime in the upcoming episodes, might in some in way use it against Flint - so in the future it would be the levels of betrayal to which he's willing to go to, if he would be willing to use that against Flint, one could say Flint has is no angel in that department himself, but it could be because Flint knows what HE would do and he knows Silver would go every bit as far, sometimes seeing someone who would be as bad as you can be very scary. Flint killed 7 men to hide the treasure, how many might Silver kill to find it? Silver might scare both Flint and Billy but they aren't necessarily fearful for the same reasons.