WIP: add javascript tokenizer

[jakubzitny]

Hey,

I am working (in coordination with @crista) on a javascript tokenizer for SourcererCC. This is the first working concept written in node.js, so we can test it out and communicate better.

This is a Work In Progress.
I don't think this is good enough to be merged yet, we should think about better way to manage the tokenizer "modules" in SourcererCC. It would be great to be able to switch between tokenizers in the the .properties config and to have (at least roughly) the same api. What do you think? Maybe

I will post updated version with better preprocessing of JS files and a script for generically running the tokenization.

[saini]

let's specify an output file and print our results to that file. Sometimes developers might want to debug using plain old school console.log approach and so let's use console only for the debug or auxiliary info.

[saini]

Hey Jakub,
Interesting profile pic :).
Thanks for submitting a PR. I went through the code and it looks good to me to start from here. I have provided a few code review comments and once you can address them we can merge your code into the master.

we should think about better way to manage the tokenizer "modules" in SourcererCC. It would be great to be able to switch between tokenizers in the the .properties config and to have (at least roughly) the same api. What do you think? Maybe

Yes, we should do that. For now, let's note it down in a new ticket.

I will post updated version with better preprocessing of JS files and a script for generically running the tokenization.
Sure. How about first we work on merging the existing code and test it. After this we can work on the newer better versions?

Thanks again for the great work
Vaibhav

[jakubzitny]

Hi @saini, thanks for the code-review, I agree with most of the comments. I'll polish the code a bit, update the MR and also file some ideas and issues I had with SourcererCC in general.

[saini]

Sounds good 👍