[question] why not droping duplicates such as save_new_companies function?

[keontang]

qlib/scripts/data_collector/index.py

Line 228 in 7d66e4b

new_df = pd.concat([new_df, _tmp_df], sort=False)

shall we do it like this:
new_df = new_df.drop_duplicates([self.SYMBOL_FIELD_NAME]) ????

[SunsetWolf]

Hi, @keontang
Thanks for your attention to qlib. parse_instruments() does not droping duplicates as expected.
In its context, repeating symbols represent different validity intervals, which are necessary for the correctness of the historical data of the index components, rather than redundant data.