diff --git a/README.md b/README.md index 373130f..ce51a55 100644 --- a/README.md +++ b/README.md @@ -1,64 +1,58 @@ - # Dirmap [English](./README_EN.md) -一个高级web目录扫描工具，功能将会强于DirBuster、Dirsearch、cansina、御剑 +An advanced web directory scanning tool, more powerful than DirBuster, Dirsearch, cansina, and Yu Jian ![dirmap](doc/dirmap.png) -# 需求分析 +# Requirements Analysis -经过大量调研，总结一个优秀的web目录扫描工具至少具备以下功能： +After extensive research, an excellent web directory scanning tool should have at least the following features: -- 并发引擎 -- 能使用字典 -- 能纯爆破 -- 能爬取页面动态生成字典 -- 能fuzz扫描 -- 自定义请求 -- 自定义响应结果处理... +- Concurrent engine +- Dictionary support +- Pure brute force capability +- Page crawling for dynamic dictionary generation +- Fuzz scanning capability +- Custom requests +- Custom response result processing... -那么接下来看看Dirmap的**特点**吧 +So let's take a look at Dirmap's **features** -# 功能特点 +# Features -1. 支持n个target\*n个payload并发 -2. 支持递归扫描 -3. 支持自定义需要递归扫描的状态码 -4. 支持(单|多)字典扫描 -5. 支持自定义字符集爆破 -6. 支持爬虫动态字典扫描 -7. 支持自定义标签fuzz目标url -8. 自定义请求User-Agent -9. 自定义请求随机延时 -10. 自定义请求超时时间 -11. 自定义请求代理 -12. 自定义正则表达式匹配假性404页面 -13. 自定义要处理的响应状态码 -14. 自定义跳过大小为x的页面 -15. 自定义显示content-type -16. 自定义显示页面大小 -17. 按域名去重复保存结果 +1. Supports n targets * n payloads concurrency +2. Supports recursive scanning +3. Supports custom status codes for recursive scanning +4. Supports (single|multiple) dictionary scanning +5. Supports custom character set brute force +6. Supports crawler dynamic dictionary scanning +7. Supports custom label fuzzing for target URLs +8. Custom request User-Agent +9. Custom request random delay +10. Custom request timeout +11. Custom request proxy +12. Custom regular expression matching for fake 404 pages +13. Custom response status codes to handle +14. Custom skip pages of size x +15. Custom display content-type +16. Custom display page size +17. Deduplicate and save results by domain -# 使用方法 +# Usage -## 环境准备 +## Environment Setup ```shell -git clone https://github.com/H4ckForJob/dirmap.git && cd dirmap && python3 -m pip install -r requirement.txt +git clone && cd dirmap && python3 -m pip install -r requirement.txt ``` -## 快速使用 +## Quick Start -### 输入目标 +### Input Target -单目标，默认为http +Single target, default is http ```shell python3 dirmap.py -i https://target.com -lcf @@ -68,307 +62,265 @@ python3 dirmap.py -i https://target.com -lcf python3 dirmap.py -i 192.168.1.1 -lcf ``` -子网（CIDR格式） +Subnet (CIDR format) ```shell python3 dirmap.py -i 192.168.1.0/24 -lcf ``` -网络范围 +Network range ```shell python3 dirmap.py -i 192.168.1.1-192.168.1.100 -lcf ``` -### 文件读取 +### File Reading ```shell python3 dirmap.py -iF targets.txt -lcf ``` -`targets.txt`中支持上述格式 +`targets.txt` supports the above formats -### 结果保存 +### Result Saving -1. 结果将自动保存在项目根目录下的`output`文件夹中 -2. 每一个目标生成一个txt，命名格式为`目标域名.txt` -3. 结果自动去重复，不用担心产生大量冗余 +1. Results will be automatically saved in the `output` folder in the project root directory +2. Each target generates a txt file, named in the format `target_domain.txt` +3. Results are automatically deduplicated, no need to worry about large amounts of redundancy -## 高级使用 +## Advanced Usage -自定义dirmap配置，开始探索dirmap高级功能 +Customize dirmap configuration to explore advanced features of dirmap -暂时采用加载配置文件的方式进行详细配置，**不支持使用命令行参数进行详细配置**！ +Currently, configuration is done by loading configuration files, **detailed configuration via command line parameters is not supported**! -编辑项目根目录下的`dirmap.conf`，进行配置 +Edit `dirmap.conf` in the project root directory to configure -`dirmap.conf`配置详解 +`dirmap.conf` Configuration Explanation ``` -#递归扫描处理配置 +# Recursive scanning processing configuration [RecursiveScan] -#是否开启递归扫描:关闭:0;开启:1 +# Enable recursive scanning: Disabled:0; Enabled:1 conf.recursive_scan = 0 -#遇到这些状态码，开启递归扫描。默认配置[301,403] +# Enable recursive scanning when encountering these status codes. Default configuration [301,403] conf.recursive_status_code = [301,403] -#URL超过这个长度就退出扫描 +# Exit scanning when URL exceeds this length conf.recursive_scan_max_url_length = 60 -#这些后缀名不递归扫 +# Do not recursively scan these extensions conf.recursive_blacklist_exts = ["html",'htm','shtml','png','jpg','webp','bmp','js','css','pdf','ini','mp3','mp4'] -#设置排除扫描的目录。默认配置空。其他配置：e.g:['/test1','/test2'] +# Set excluded scan directories. Default configuration is empty. Other configuration: e.g:['/test1','/test2'] #conf.exclude_subdirs = ['/test1','/test2'] conf.exclude_subdirs = "" -#扫描模式处理配置(4个模式，1次只能选择1个) +# Scanning mode processing configuration (4 modes, can only select 1 at a time) [ScanModeHandler] -#字典模式:关闭:0;单字典:1;多字典:2 +# Dictionary mode: Disabled:0; Single dictionary:1; Multiple dictionaries:2 conf.dict_mode = 1 -#单字典模式的路径 +# Path for single dictionary mode conf.dict_mode_load_single_dict = "dict_mode_dict.txt" -#多字典模式的路径，默认配置dictmult +# Path for multiple dictionary mode, default configuration is dictmult conf.dict_mode_load_mult_dict = "dictmult" -#爆破模式:关闭:0;开启:1 +# Brute force mode: Disabled:0; Enabled:1 conf.blast_mode = 0 -#生成字典最小长度。默认配置3 +# Minimum length for generated dictionary. Default configuration is 3 conf.blast_mode_min = 3 -#生成字典最大长度。默认配置3 +# Maximum length for generated dictionary. Default configuration is 3 conf.blast_mode_max = 3 -#默认字符集:a-z。暂未使用。 +# Default character set: a-z. Not yet used. conf.blast_mode_az = "abcdefghijklmnopqrstuvwxyz" -#默认字符集:0-9。暂未使用。 +# Default character set: 0-9. Not yet used. conf.blast_mode_num = "0123456789" -#自定义字符集。默认配置"abc"。使用abc构造字典 +# Custom character set. Default configuration is "abc". Use abc to construct dictionary conf.blast_mode_custom_charset = "abc" -#自定义继续字符集。默认配置空。 +# Custom resume character set. Default configuration is empty. conf.blast_mode_resume_charset = "" -#爬虫模式:关闭:0;开启:1 +# Crawler mode: Disabled:0; Enabled:1 conf.crawl_mode = 0 -#用于生成动态敏感文件payload的后缀字典 +# Suffix dictionary for generating dynamic sensitive file payloads conf.crawl_mode_dynamic_fuzz_suffix = "crawl_mode_suffix.txt" -#解析robots.txt文件。暂未实现。 +# Parse robots.txt file. Not yet implemented. conf.crawl_mode_parse_robots = 0 -#解析html页面的xpath表达式 +# Xpath expression for parsing html pages conf.crawl_mode_parse_html = "//*/@href | //*/@src | //form/@action" -#是否进行动态爬虫字典生成。默认配置1，开启爬虫动态字典生成。其他配置：e.g:关闭:0;开启:1 +# Whether to perform dynamic crawler dictionary generation. Default configuration is 1, enable crawler dynamic dictionary generation. Other configuration: e.g: Disabled:0; Enabled:1 conf.crawl_mode_dynamic_fuzz = 1 -#Fuzz模式:关闭:0;单字典:1;多字典:2 +# Fuzz mode: Disabled:0; Single dictionary:1; Multiple dictionaries:2 conf.fuzz_mode = 0 -#单字典模式的路径。 +# Path for single dictionary mode. conf.fuzz_mode_load_single_dict = "fuzz_mode_dir.txt" -#多字典模式的路径。默认配置:fuzzmult +# Path for multiple dictionary mode. Default configuration is: fuzzmult conf.fuzz_mode_load_mult_dict = "fuzzmult" -#设置fuzz标签。默认配置{dir}。使用{dir}标签当成字典插入点，将http://target.com/{dir}.php替换成http://target.com/字典中的每一行.php。其他配置：e.g:{dir};{ext} +# Set fuzz label. Default configuration is {dir}. Use {dir} label as dictionary insertion point, replace http://target.com/{dir}.php with http://target.com/each line in dictionary.php. Other configuration: e.g:{dir};{ext} #conf.fuzz_mode_label = "{ext}" conf.fuzz_mode_label = "{dir}" -#处理payload配置。暂未实现。 +# Payload processing configuration. Not yet implemented. [PayloadHandler] -#处理请求配置 +# Request processing configuration [RequestHandler] -#自定义请求头。默认配置空。其他配置：e.g:test1=test1,test2=test2 +# Custom request headers. Default configuration is empty. Other configuration: e.g:test1=test1,test2=test2 #conf.request_headers = "test1=test1,test2=test2" conf.request_headers = "" -#自定义请求User-Agent。默认配置chrome的ua。 +# Custom request User-Agent. Default configuration is chrome's ua. conf.request_header_ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" -#自定义请求cookie。默认配置空，不设置cookie。其他配置e.g:cookie1=cookie1; cookie2=cookie2; +# Custom request cookie. Default configuration is empty, no cookie set. Other configuration e.g:cookie1=cookie1; cookie2=cookie2; #conf.request_header_cookie = "cookie1=cookie1; cookie2=cookie2" conf.request_header_cookie = "" -#自定义401认证。暂未实现。因为自定义请求头功能可满足该需求(懒XD) +# Custom 401 authentication. Not yet implemented. Because custom request header function can meet this requirement (lazy XD) conf.request_header_401_auth = "" -#自定义请求方法。默认配置get方法。其他配置：e.g:get;head +# Custom request method. Default configuration is get method. Other configuration: e.g:get;head #conf.request_method = "head" conf.request_method = "get" -#自定义每个请求超时时间。默认配置3秒。 +# Custom timeout for each request. Default configuration is 3 seconds. conf.request_timeout = 3 -#随机延迟(0-x)秒发送请求。参数必须是整数。默认配置0秒，无延迟。 +# Random delay (0-x) seconds to send requests. Parameter must be integer. Default configuration is 0 seconds, no delay. conf.request_delay = 0 -#自定义单个目标，请求协程线程数。默认配置30线程 +# Custom number of request coroutine threads per target. Default configuration is 30 threads conf.request_limit = 30 -#自定义最大重试次数。暂未实现。 +# Custom maximum retry count. Not yet implemented. conf.request_max_retries = 1 -#设置持久连接。是否使用session()。暂未实现。 +# Set persistent connection. Whether to use session(). Not yet implemented. conf.request_persistent_connect = 0 -#302重定向。默认False，不重定向。其他配置：e.g:True;False +# 302 redirection. Default False, no redirection. Other configuration: e.g:True;False conf.redirection_302 = False -#payload后添加后缀。默认空，扫描时，不添加后缀。其他配置：e.g:txt;php;asp;jsp +# Add suffix after payload. Default is empty, no suffix added during scanning. Other configuration: e.g:txt;php;asp;jsp #conf.file_extension = "txt" conf.file_extension = "" -#处理响应配置 +# Response processing configuration [ResponseHandler] -#设置要记录的响应状态。默认配置[200]，记录200状态码。其他配置：e.g:[200,403,301] +# Set response statuses to record. Default configuration is [200], record 200 status code. Other configuration: e.g:[200,403,301] #conf.response_status_code = [200,403,301] conf.response_status_code = [200] -#是否记录content-type响应头。默认配置1记录 +# Whether to record content-type response header. Default configuration is 1 to record #conf.response_header_content_type = 0 conf.response_header_content_type = 1 -#是否记录页面大小。默认配置1记录 +# Whether to record page size. Default configuration is 1 to record #conf.response_size = 0 conf.response_size = 1 -#是否自动检测404页面。默认配置True，开启自动检测404.其他配置参考e.g:True;False +# Whether to automatically detect 404 pages. Default configuration is True, enable automatic 404 detection. Other configuration reference e.g:True;False #conf.auto_check_404_page = False conf.auto_check_404_page = True -#自定义匹配503页面正则。暂未实现。感觉用不着，可能要废弃。 +# Custom regular expression to match 503 pages. Not yet implemented. Feels unnecessary, might be deprecated. #conf.custom_503_page = "page 503" conf.custom_503_page = "" -#自定义正则表达式，匹配页面内容 +# Custom regular expression to match page content #conf.custom_response_page = "([0-9]){3}([a-z]){3}test" conf.custom_response_page = "" -#跳过显示页面大小为x的页面，若不设置，请配置成"None"，默认配置“None”。其他大小配置参考e.g:None;0b;1k;1m +# Skip displaying pages of size x, if not set, please configure as "None", default configuration is "None". Other size configuration reference e.g:None;0b;1k;1m #conf.skip_size = "0b" conf.skip_size = "None" -#代理选项 +# Proxy options [ProxyHandler] -#代理配置。默认设置“None”，不开启代理。其他配置e.g:{"http":"http://127.0.0.1:8080","https":"https://127.0.0.1:8080"} +# Proxy configuration. Default is "None", no proxy enabled. Other configuration e.g:{"http":"http://127.0.0.1:8080","https":"https://127.0.0.1:8080"} #conf.proxy_server = {"http":"http://127.0.0.1:8080","https":"https://127.0.0.1:8080"} conf.proxy_server = None -#Debug选项 +# Debug options [DebugMode] -#打印payloads并退出 +# Print payloads and exit conf.debug = 0 -#update选项 +# Update options [CheckUpdate] -#github获取更新。暂未实现。 +# Get updates from github. Not yet implemented. conf.update = 0 ``` # TODO -- [x] 命令行参数解析全局初始化 -- [x] engine初始化 - - [x] 设置线程数 -- [x] target初始化 - - [x] 自动解析处理输入格式( -i,inputTarget) +- [x] Command line argument parsing global initialization +- [x] engine initialization + - [x] Set thread count +- [x] target initialization + - [x] Automatically parse and handle input format (-i,inputTarget) - [x] IP - [x] Domain - [x] URL - [x] IP/MASK - [x] IP Start-End - - [x] 文件读入(-iF,inputLocalFile) -- [ ] bruter初始化 - - [ ] 加载配置方式() - - [ ] 读取命令行参数值 - - [x] 读取配置文件(-lcf,loadConfigFile) - - [x] 递归模式选项(RecursiveScan) - - [x] 递归扫描(-rs,recursive_scan) - - [x] 需要递归的状态码(-rd,recursive_status_code) - - [x] 排除某些目录(-es,exclude_subdirs) - - [ ] 扫描模式选项(ScanModeHandler) - - [x] 字典模式(-dm,dict_mode) - - [x] 加载单个字典(-dmlsd,dict_mode_load_single_dict) - - [x] 加载多个字典(-dmlmd,dict_mode_load_mult_dict) - - [ ] 爆破模式(-bm,blast_mode) - - [x] 爆破目录长度范围(必选) - - [x] 最小长度(-bmmin,blast_mode_min) - - [x] 最大长度(-bmmax,blast_mode_max) - - [ ] 基于默认字符集 - - [ ] 基于a-z - - [ ] 基于0-9 - - [x] 基于自定义字符集(-bmcc,blast_mode_custom_charset) - - [x] 断点续生成payload(-bmrc,blast_mode_resume_charset) - - [ ] 爬虫模式(-cm,crawl_mode) - - [x] 自定义解析标签(-cmph,crawl_mode_parse_html)(a:href,img:src,form:action,script:src,iframe:src,div:src,frame:src,embed:src) - - [ ] 解析robots.txt(-cmpr,crawl_mode_parse_robots) - - [x] 爬虫类动态fuzz扫描(-cmdf,crawl_mode_dynamic_fuzz) - - [x] fuzz模式(-fm,fuzz_mode) - - [x] fuzz单个字典(-fmlsd,fuzz_mode_load_single_dict) - - [x] fuzz多个字典(-fmlmd,fuzz_mode_load_mult_dict) - - [x] fuzz标签(-fml,fuzz_mode_label) - - [ ] 请求优化选项(RequestHandler) - - [x] 自定义请求超时(-rt,request_timeout) - - [x] 自定义请求延时(-rd,request_delay) - - [x] 限制单个目标主机协程数扫描(-rl,request_limit) - - [ ] 限制重试次数(-rmr,request_max_retries) - - [ ] http持久连接(-rpc,request_persistent_connect) - - [x] 自定义请求方法(-rm,request_method)(get、head) - - [x] 302状态处理(-r3,redirection_302)(是否重定向) - - [x] 自定义header - - [x] 自定义其他header(-rh,request_headers)(解决需要401认证) - - [x] 自定义ua(-rhua,request_header_ua) - - [x] 自定义cookie(-rhc,request_header_cookie) - - [ ] 字典处理选项(PayloadHandler) - - [ ] 字典处理(payload修改-去斜杠) - - [ ] 字典处理(payload修改-首字符加斜杠) - - [ ] 字典处理(payload修改-单词首字母大写) - - [ ] 字典处理(payload修改-去扩展) - - [ ] 字典处理(payload修改-去除非字母数字) - - [ ] 响应结果处理模块(ResponseHandler) - - [x] 跳过大小为x字节的文件(-ss,skip_size) - - [x] 自动检测404页面(-ac4p,auto_check_404_page) - - [ ] 自定义503页面(-c5p,custom_503_page) - - [ ] 自定义正则匹配响应内容并进行某种操作 - - [x] 自定义正则匹配响应(-crp,custom_response_page) - - [ ] 某种操作(暂时未定义) - - [x] 输出结果为自定义状态码(-rsc,response_status_code) - - [x] 输出payload为完整路径(默认输出完成url) - - [x] 输出结果展示content-type - - [x] 自动去重复保存结果 - - [ ] 状态处理模块(StatusHandler) - - [ ] 状态显示(等待开始、进行中、暂停中、异常、完成) - - [x] 进度显示 - - [ ] 状态控制(开始、暂停、继续、停止) - - [ ] 续扫模块(暂未配置) - - [ ] 断点续扫 - - [ ] 选行续扫 - - [ ] 日志记录模块(ScanLogHandler) - - [ ] 扫描日志 - - [ ] 错误日志 - - [ ] 代理模块(ProxyHandler) - - [x] 单个代理(-ps,proxy_server) - - [ ] 代理池 - - [x] 调试模式选项(DebugMode) - - [x] debug(--debug) - - [ ] 检查更新选项(CheckUpdate) - - [ ] update(--update) - -# 默认字典文件 - -字典文件存放在项目根目录中的`data`文件夹中 - -1. dict_mode_dict.txt “字典模式”字典，使用dirsearch默认字典 -2. crawl_mode_suffix.txt “爬虫模式”字典，使用FileSensor默认字典 -3. fuzz_mode_dir.txt “fuzz模式”字典，使用DirBuster默认字典 -4. fuzz_mode_ext.txt “fuzz模式”字典，使用常见后缀制作的字典 -5. dictmult 该目录为“字典模式”默认多字典文件夹，包含：BAK.min.txt(备份文件小字典)，BAK.txt(备份文件大字典)，LEAKS.txt(信息泄露文件字典) -6. fuzzmult 该目录为“fuzz模式”默认多字典文件夹，包含：fuzz_mode_dir.txt(默认目录字典)，fuzz_mode_ext.txt(默认后缀字典) - -# 已知缺陷 - -1. “爬虫模式”只爬取了目标的当前页面，用于生成动态字典。项目将来会将“爬虫模块”与“生成动态字典功能”分离。 -2. 关于bruter.py第517行`bar.log.start()`出错。解决方案：请安装progressbar2。卸载progressbar。防止导入同名模块。感谢某位表哥提醒。 - -```shell -执行命令： -python3 -m pip uninstall progressbar -python3 -m pip install progressbar2 -``` - -# 维护工作 - -1. 若使用过程中出现问题，欢迎发issue -2. 本项目正在维护，未来将会有新的功能加入，具体参照“TODO”列表，未打勾项 - -# 致谢声明 - -dirmap在编写过程中，借鉴了大量的优秀开源项目的模式与思想，特此说明并表示感谢。 - -- [Sqlmap](https://github.com/sqlmapproject/sqlmap) -- [POC-T](https://github.com/Xyntax/POC-T) -- [Saucerframe](https://github.com/saucer-man/saucerframe) -- [gwhatweb](https://github.com/boy-hack/gwhatweb) -- [dirsearch](https://github.com/maurosoria/dirsearch) + - [x] File input (-iF,inputLocalFile) +- [ ] bruter initialization + - [ ] Configuration loading method() + - [ ] Read command line parameter values + - [x] Read configuration file (-lcf,loadConfigFile) + - [x] Recursive mode options (RecursiveScan) + - [x] Recursive scanning (-rs,recursive_scan) + - [x] Status codes requiring recursion (-rd,recursive_status_code) + - [x] Exclude certain directories (-es,exclude_subdirs) + - [ ] Scanning mode options (ScanModeHandler) + - [x] Dictionary mode (-dm,dict_mode) + - [x] Load single dictionary (-dmlsd,dict_mode_load_single_dict) + - [x] Load multiple dictionaries (-dmlmd,dict_mode_load_mult_dict) + - [ ] Brute force mode (-bm,blast_mode) + - [x] Brute force directory length range (required) + - [x] Minimum length (-bmmin,blast_mode_min) + - [x] Maximum length (-bmmax,blast_mode_max) + - [ ] Based on default character sets + - [ ] Based on a-z + - [ ] Based on 0-9 + - [x] Based on custom character set (-bmcc,blast_mode_custom_charset) + - [x] Resume payload generation (-bmrc,blast_mode_resume_charset) + - [ ] Crawler mode (-cm,crawl_mode) + - [x] Custom parsing tags (-cmph,crawl_mode_parse_html)(a:href,img:src,form:action,script:src,iframe:src,div:src,frame:src,embed:src) + - [ ] Parse robots.txt (-cmpr,crawl_mode_parse_robots) + - [x] Crawler dynamic fuzz scanning (-cmdf,crawl_mode_dynamic_fuzz) + - [x] Fuzz mode (-fm,fuzz_mode) + - [x] Fuzz single dictionary (-fmlsd,fuzz_mode_load_single_dict) + - [x] Fuzz multiple dictionaries (-fmlmd,fuzz_mode_load_mult_dict) + - [x] Fuzz label (-fml,fuzz_mode_label) + - [ ] Request optimization options (RequestHandler) + - [x] Custom request timeout (-rt,request_timeout) + - [x] Custom request delay (-rd,request_delay) + - [x] Limit coroutine threads per target host (-rl,request_limit) + - [ ] Limit retry count (-rmr,request_max_retries) + - [ ] HTTP persistent connection (-rpc,request_persistent_connect) + - [x] Custom request method (-rm,request_method)(get,head) + - [x] 302 status handling (-r3,redirection_302)(whether to redirect) + - [x] Custom header + - [x] Custom other headers (-rh,request_headers)(solve 401 authentication) + - [x] Custom ua (-rhua,request_header_ua) + - [x] Custom cookie (-rhc,request_header_cookie) + - [ ] Dictionary processing options (PayloadHandler) + - [ ] Dictionary processing (payload modification - remove slash) + - [ ] Dictionary processing (payload modification - add leading slash) + - [ ] Dictionary processing (payload modification - capitalize first letter) + - [ ] Dictionary processing (payload modification - remove extensions) + - [ ] Dictionary processing (payload modification - remove non-alphanumeric) + - [ ] Response result processing module (ResponseHandler) + - [x] Response filtering (response status filtering) + - [x] Response filtering (response header filtering) + - [x] Response filtering (response size filtering) + - [x] Response filtering (automatic 404 detection) + - [x] Response filtering (custom response matching) + - [x] Response processing (domain deduplication) + - [x] Response processing (save results) + - [ ] Plugin system + - [ ] Scan plugin interface + - [ ] Response processing plugin interface + - [ ] Debug options + - [x] Debug mode (-debug,debug) + - [ ] Update options + - [ ] Update function (-u,update) + +# Star History + + +# Contributing + +1. Fork this repository +2. Create your feature branch (`git checkout -b feature/AmazingFeature`) +3. Commit your changes (`git commit -m 'Add some AmazingFeature'`) +4. Push to the branch (`git push origin feature/AmazingFeature`) +5. Open a Pull Request + +# Acknowledgments + +Thanks to the following tools for reference: +- [DirBuster](https://sourceforge.net/projects/dirbuster/) +- [Dirsearch](https://github.com/maurosoria/dirsearch) - [cansina](https://github.com/deibit/cansina) -- [weakfilescan](https://github.com/ring04h/weakfilescan) -- [FileSensor](https://github.com/Xyntax/FileSensor) -- [BBscan](https://github.com/lijiejie/BBScan) -- [werdy](https://github.com/derv82/werdy) - -# 联系作者 - -mail: xxlin.ujs@qq.com - -![donate](doc/donate.jpg) \ No newline at end of file +- [御剑](http://www.webcache.com/soft/) \ No newline at end of file diff --git a/README_EN.md b/README_EN.md index f8ad54c..f2b7ed5 100644 --- a/README_EN.md +++ b/README_EN.md @@ -1,12 +1,6 @@ - # Dirmap -[中文](./README.md) +[Chinese](./README.md) An advanced web directory scanning tool that will be more powerful than DirBuster, Dirsearch, cansina, and Yu Jian @@ -51,7 +45,7 @@ Then take a look at the **features** of Dirmap. ## Environment and download ```shell -git clone https://github.com/H4ckForJob/dirmap.git && cd dirmap && python3 -m pip install -r requirement.txt +git clone && cd dirmap && python3 -m pip install -r requirement.txt ``` ## Quick use @@ -234,6 +228,5 @@ In the process of writing dirmap, I borrowed a lot of models and ideas from exce # Contact -mail: xxlin.ujs@qq.com ![donate](doc/donate.jpg) \ No newline at end of file diff --git a/dirmap.py b/dirmap.py index 20c04c4..e4f1220 100644 --- a/dirmap.py +++ b/dirmap.py @@ -1,20 +1,13 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:59 -@LastEditTime: 2023-07-25 15:56:04 -''' - import os import sys from gevent import monkey monkey.patch_all() from lib.controller.engine import run -from lib.core.common import banner, outputscreen, setPaths +from lib.core.common import outputscreen, setPaths from lib.core.data import cmdLineOptions, conf, paths from lib.core.option import initOptions from lib.parse.cmdline import cmdLineParser @@ -30,8 +23,7 @@ def main(): if sys.version_info < (3, 8): outputscreen.error("Sorry, dirmap requires Python 3.8 or higher\n") sys.exit(1) - # anyway output thr banner information - banner() + # set paths of project paths.ROOT_PATH = os.getcwd() diff --git a/fix_syntax.py b/fix_syntax.py new file mode 100644 index 0000000..974e2b5 --- /dev/null +++ b/fix_syntax.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +import os +import re + +def fix_syntax_errors(directory): + """Fix unterminated triple quotes in Python files""" + for root, dirs, files in os.walk(directory): + # Skip thirdlib directory for now + if 'thirdlib' in root: + continue + + for file in files: + if file.endswith('.py'): + filepath = os.path.join(root, file) + try: + with open(filepath, 'r', encoding='utf-8') as f: + content = f.read() + + # Remove lines that are just triple quotes at the start + lines = content.split('\n') + new_lines = [] + + for i, line in enumerate(lines): + # If it's just triple quotes and appears early in the file (first 10 lines) + if line.strip() == "'''" and i < 10: + continue + new_lines.append(line) + + new_content = '\n'.join(new_lines) + + if new_content != content: + with open(filepath, 'w', encoding='utf-8') as f: + f.write(new_content) + print(f"Fixed: {filepath}") + + except Exception as e: + print(f"Error processing {filepath}: {e}") + +if __name__ == "__main__": + fix_syntax_errors("/home/sondt/dirmap") \ No newline at end of file diff --git a/lib/__init__.py b/lib/__init__.py index 2b1d7a4..7847780 100644 --- a/lib/__init__.py +++ b/lib/__init__.py @@ -1,9 +1,3 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 16:11:27 -@LastEditTime: 2019-04-11 20:04:09 -''' diff --git a/lib/controller/__init__.py b/lib/controller/__init__.py index 4e4c996..6f7b18c 100644 --- a/lib/controller/__init__.py +++ b/lib/controller/__init__.py @@ -1,9 +1,2 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- - -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 20:11:08 -@LastEditTime: 2019-04-11 20:11:12 -''' diff --git a/lib/controller/bruter.py b/lib/controller/bruter.py index e6dc51c..f9e57f1 100644 --- a/lib/controller/bruter.py +++ b/lib/controller/bruter.py @@ -1,14 +1,7 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-03-14 09:49:05 -@LastEditTime: 2023-07-25 16:32:33 -''' - -import configparser +import ast import hashlib import os import random @@ -28,42 +21,42 @@ from lib.utils.config import ConfigFileParser from lib.plugins.inspector import Inspector -#防止ssl未校验时出现提示信息 +# Prevent SSL warning messages when verification is disabled requests.packages.urllib3.disable_warnings() -#dict_mode的payloads +# payloads for dict_mode payloads.dict_mode_dict = set() -#crawl_mode的payloads +# payloads for crawl_mode payloads.crawl_mode_dynamic_fuzz_temp_dict = set() payloads.similar_urls_set = set() payloads.crawl_mode_dynamic_fuzz_dict = list() -#blast_mode的payload +# payload for blast_mode payloads.blast_mode_custom_charset_dict = list() -#fuzz_mode的payload +# payload for fuzz_mode payloads.fuzz_mode_dict = list() -#创建all_tasks队列 +# create all_tasks queue tasks.all_task = Queue() tasks.task_length = 0 tasks.task_count = 0 -#创建crawl_tasks队列 +# create crawl_tasks queue tasks.crawl_task = Queue() -#假性404页面md5列表 +# fake 404 page md5 list conf.autodiscriminator_md5 = set() bar.log = progressbar.ProgressBar() def saveResults(domain,msg): ''' - @description: 结果保存，以"域名.txt"命名，url去重复 - @param {domain:域名,msg:保存的信息} + @description: Save results, named as "domain.txt", deduplicate URLs + @param {domain: domain name, msg: information to save} @return: null ''' filename = domain +'.txt' conf.output_path = os.path.join(paths.OUTPUT_PATH, filename) - #判断文件是否存在，若不存在则创建该文件 + # Check if file exists, create if not if not os.path.exists(conf.output_path): with open(conf.output_path,'w+') as temp: pass @@ -74,102 +67,111 @@ def saveResults(domain,msg): else: result_file.write(msg+'\n') +def _safe_eval(config_string): + """ + Safely evaluate configuration values. + Uses ast.literal_eval for security, with fallback to string values. + """ + if not config_string: + return "" + try: + return ast.literal_eval(config_string) + except (ValueError, SyntaxError): + return config_string + def loadConf(): ''' - @description: 加载扫描配置(以后将使用参数，而非从文件加载) + @description: Load scanning configuration (will use parameters instead of loading from file in the future) @param {type} @return: ''' - conf.recursive_scan = eval(ConfigFileParser().recursive_scan()) - conf.recursive_scan_max_url_length = eval(ConfigFileParser().recursive_scan_max_url_length()) - conf.recursive_status_code = eval(ConfigFileParser().recursive_status_code()) - conf.recursive_blacklist_exts = eval(ConfigFileParser().recursive_blacklist_exts()) - conf.exclude_subdirs = eval(ConfigFileParser().exclude_subdirs()) - - conf.dict_mode = eval(ConfigFileParser().dict_mode()) - conf.dict_mode_load_single_dict = os.path.join(paths.DATA_PATH,eval(ConfigFileParser().dict_mode_load_single_dict())) - conf.dict_mode_load_mult_dict = os.path.join(paths.DATA_PATH,eval(ConfigFileParser().dict_mode_load_mult_dict())) - conf.blast_mode = eval(ConfigFileParser().blast_mode()) - conf.blast_mode_min = eval(ConfigFileParser().blast_mode_min()) - conf.blast_mode_max = eval(ConfigFileParser().blast_mode_max()) - conf.blast_mode_az = eval(ConfigFileParser().blast_mode_az()) - conf.blast_mode_num = eval(ConfigFileParser().blast_mode_num()) - conf.blast_mode_custom_charset = eval(ConfigFileParser().blast_mode_custom_charset()) - conf.blast_mode_resume_charset = eval(ConfigFileParser().blast_mode_resume_charset()) - conf.crawl_mode = eval(ConfigFileParser().crawl_mode()) - conf.crawl_mode_dynamic_fuzz_suffix = eval(ConfigFileParser().crawl_mode_dynamic_fuzz_suffix()) - conf.crawl_mode_parse_robots = eval(ConfigFileParser().crawl_mode_parse_robots()) - conf.crawl_mode_parse_html = eval(ConfigFileParser().crawl_mode_parse_html()) - conf.crawl_mode_dynamic_fuzz = eval(ConfigFileParser().crawl_mode_dynamic_fuzz()) - conf.fuzz_mode = eval(ConfigFileParser().fuzz_mode()) - conf.fuzz_mode_load_single_dict = os.path.join(paths.DATA_PATH,eval(ConfigFileParser().fuzz_mode_load_single_dict())) - conf.fuzz_mode_load_mult_dict = os.path.join(paths.DATA_PATH,eval(ConfigFileParser().fuzz_mode_load_mult_dict())) - conf.fuzz_mode_label = eval(ConfigFileParser().fuzz_mode_label()) - - conf.request_headers = eval(ConfigFileParser().request_headers()) - conf.request_header_ua = eval(ConfigFileParser().request_header_ua()) - conf.request_header_cookie = eval(ConfigFileParser().request_header_cookie()) - conf.request_header_401_auth = eval(ConfigFileParser().request_header_401_auth()) - conf.request_timeout = eval(ConfigFileParser().request_timeout()) - conf.request_delay = eval(ConfigFileParser().request_delay()) - conf.request_limit = eval(ConfigFileParser().request_limit()) - conf.request_max_retries = eval(ConfigFileParser().request_max_retries()) - conf.request_persistent_connect = eval(ConfigFileParser().request_persistent_connect()) - conf.request_method = eval(ConfigFileParser().request_method()) - conf.redirection_302 = eval(ConfigFileParser().redirection_302()) - conf.file_extension = eval(ConfigFileParser().file_extension()) - - conf.response_status_code = eval(ConfigFileParser().response_status_code()) - conf.response_header_content_type = eval(ConfigFileParser().response_header_content_type()) - conf.response_size = eval(ConfigFileParser().response_size()) - conf.auto_check_404_page = eval(ConfigFileParser().auto_check_404_page()) - conf.custom_503_page = eval(ConfigFileParser().custom_503_page()) - conf.custom_response_page = eval(ConfigFileParser().custom_response_page()) - conf.skip_size = eval(ConfigFileParser().skip_size()) - - conf.proxy_server = eval(ConfigFileParser().proxy_server()) - - conf.debug = eval(ConfigFileParser().debug()) - conf.update = eval(ConfigFileParser().update()) + conf.recursive_scan = _safe_eval(ConfigFileParser().recursive_scan()) + conf.recursive_scan_max_url_length = _safe_eval(ConfigFileParser().recursive_scan_max_url_length()) + conf.recursive_status_code = _safe_eval(ConfigFileParser().recursive_status_code()) + conf.recursive_blacklist_exts = _safe_eval(ConfigFileParser().recursive_blacklist_exts()) + conf.exclude_subdirs = _safe_eval(ConfigFileParser().exclude_subdirs()) + + conf.dict_mode = _safe_eval(ConfigFileParser().dict_mode()) + conf.dict_mode_load_single_dict = os.path.join(paths.DATA_PATH,_safe_eval(ConfigFileParser().dict_mode_load_single_dict())) + conf.dict_mode_load_mult_dict = os.path.join(paths.DATA_PATH,_safe_eval(ConfigFileParser().dict_mode_load_mult_dict())) + conf.blast_mode = _safe_eval(ConfigFileParser().blast_mode()) + conf.blast_mode_min = _safe_eval(ConfigFileParser().blast_mode_min()) + conf.blast_mode_max = _safe_eval(ConfigFileParser().blast_mode_max()) + conf.blast_mode_az = _safe_eval(ConfigFileParser().blast_mode_az()) + conf.blast_mode_num = _safe_eval(ConfigFileParser().blast_mode_num()) + conf.blast_mode_custom_charset = _safe_eval(ConfigFileParser().blast_mode_custom_charset()) + conf.blast_mode_resume_charset = _safe_eval(ConfigFileParser().blast_mode_resume_charset()) + conf.crawl_mode = _safe_eval(ConfigFileParser().crawl_mode()) + conf.crawl_mode_dynamic_fuzz_suffix = _safe_eval(ConfigFileParser().crawl_mode_dynamic_fuzz_suffix()) + conf.crawl_mode_parse_robots = _safe_eval(ConfigFileParser().crawl_mode_parse_robots()) + conf.crawl_mode_parse_html = _safe_eval(ConfigFileParser().crawl_mode_parse_html()) + conf.crawl_mode_dynamic_fuzz = _safe_eval(ConfigFileParser().crawl_mode_dynamic_fuzz()) + conf.fuzz_mode = _safe_eval(ConfigFileParser().fuzz_mode()) + conf.fuzz_mode_load_single_dict = os.path.join(paths.DATA_PATH,_safe_eval(ConfigFileParser().fuzz_mode_load_single_dict())) + conf.fuzz_mode_load_mult_dict = os.path.join(paths.DATA_PATH,_safe_eval(ConfigFileParser().fuzz_mode_load_mult_dict())) + conf.fuzz_mode_label = _safe_eval(ConfigFileParser().fuzz_mode_label()) + + conf.request_headers = _safe_eval(ConfigFileParser().request_headers()) + conf.request_header_ua = _safe_eval(ConfigFileParser().request_header_ua()) + conf.request_header_cookie = _safe_eval(ConfigFileParser().request_header_cookie()) + conf.request_header_401_auth = _safe_eval(ConfigFileParser().request_header_401_auth()) + conf.request_timeout = _safe_eval(ConfigFileParser().request_timeout()) + conf.request_delay = _safe_eval(ConfigFileParser().request_delay()) + conf.request_limit = _safe_eval(ConfigFileParser().request_limit()) + conf.request_max_retries = _safe_eval(ConfigFileParser().request_max_retries()) + conf.request_persistent_connect = _safe_eval(ConfigFileParser().request_persistent_connect()) + conf.request_method = _safe_eval(ConfigFileParser().request_method()) + conf.redirection_302 = _safe_eval(ConfigFileParser().redirection_302()) + conf.file_extension = _safe_eval(ConfigFileParser().file_extension()) + + conf.response_status_code = _safe_eval(ConfigFileParser().response_status_code()) + conf.response_header_content_type = _safe_eval(ConfigFileParser().response_header_content_type()) + conf.response_size = _safe_eval(ConfigFileParser().response_size()) + conf.auto_check_404_page = _safe_eval(ConfigFileParser().auto_check_404_page()) + conf.custom_503_page = _safe_eval(ConfigFileParser().custom_503_page()) + conf.custom_response_page = _safe_eval(ConfigFileParser().custom_response_page()) + conf.skip_size = _safe_eval(ConfigFileParser().skip_size()) + + conf.proxy_server = _safe_eval(ConfigFileParser().proxy_server()) + + conf.debug = _safe_eval(ConfigFileParser().debug()) + conf.update = _safe_eval(ConfigFileParser().update()) def recursiveScan(response_url,all_payloads): ''' - @description: 检测出一级目录后，一级目录后遍历添加所有payload，继续检测 + @description: After detecting first-level directories, traverse and add all payloads to continue detection @param {type} @return: ''' if not conf.recursive_scan: return - # 当前url后缀在黑名单内，不进行递归 + # Skip recursive if current URL extension is in blacklist if response_url.split('.')[-1].lower() in conf.recursive_blacklist_exts: return - #XXX:payloads字典要固定格式 + #XXX:payloads dictionary needs fixed format for payload in all_payloads: - #判断是否排除。若在排除的目录列表中，则排除。self.excludeSubdirs排除的列表，配置文件中，形如:/test、/test1 + # Check if excluded. If in excluded directory list, exclude it. exclude_subdirs list format in config file: /test, /test1 if payload in [directory for directory in conf.exclude_subdirs]: return - #payload拼接，处理/重复或缺失 - if response_url.endswith('/') and payload.startswith('/'): - # /重复，url和payload都有/，删去payload的/前缀 - payload = payload[1:] - elif (not response_url.endswith('/')) and (not payload.startswith('/')): - # /缺失，url和payload都不包含/，在payload前追加/ - payload = '/'+payload - #拼接payload，限制url长度，入队tasks - newpayload=response_url+payload + # Use urljoin to correctly handle URL concatenation, avoiding double slash issue + # Ensure response_url ends with /, then use urljoin + if not response_url.endswith('/'): + response_url = response_url + '/' + # Use urljoin to concatenate URLs, it automatically handles slash issues + newpayload = urllib.parse.urljoin(response_url, payload.lstrip('/')) if(len(newpayload) < int(conf.recursive_scan_max_url_length)): - tasks.all_task.put(response_url + payload) + tasks.all_task.put(newpayload) def loadSingleDict(path): ''' - @description: 添加单个字典文件 - @param {path:字典文件路径} + @description: Load single dictionary file + @param {path: dictionary file path} @return: ''' try: outputscreen.success('[+] Load dict:{}'.format(path)) - #加载文件时，使用utf-8编码，防止出现编码问题 + # Use utf-8 encoding when loading files to prevent encoding issues with open(path,encoding='utf-8') as single_file: return single_file.read().splitlines() except Exception as e: @@ -178,14 +180,14 @@ def loadSingleDict(path): def loadMultDict(path): ''' - @description: 添加多个字典文件 - @param {path:字典文件路径} + @description: Load multiple dictionary files + @param {path: dictionary file path} @return: ''' tmp_list = [] try: for file in os.listdir(path): - #FIXME:这里解决dict和fuzz模式加载多字典问题，但是loadMultDict变得臃肿，后期需要处理 + #FIXME: This solves the problem of loading multiple dictionaries in dict and fuzz modes, but makes loadMultDict bloated, needs refactoring later if conf.dict_mode and conf.fuzz_mode: outputscreen.error('[x] Can not use dict and fuzz mode at the same time!') sys.exit() @@ -200,13 +202,13 @@ def loadMultDict(path): def loadSuffix(path): ''' - @description: 添加动态爬虫字典后缀规则 + @description: Load dynamic crawler dictionary suffix rules @param {type} @return: ''' try: with open(path) as f: - #要去掉#开头的字典 + # Remove dictionary entries starting with # payloads.suffix = set(f.read().split('\n')) - {'', '#'} except Exception as e: outputscreen.error('[x] plz check file path!\n[x] error:{}'.format(e)) @@ -214,7 +216,7 @@ def loadSuffix(path): def generateCrawlDict(base_url): ''' - @description: 生成动态爬虫字典 + @description: Generate dynamic crawler dictionary @param {base_url:} @return: ''' @@ -241,7 +243,11 @@ def _splitFilename(filename): final_urls = list() for each in payloads.suffix: - new_filename = path + '/' + each.replace('{FULL}', filename) + # Use urljoin to correctly handle path concatenation, avoiding double slash issue + # Ensure path ends with / + if not path.endswith('/'): + path = path + '/' + new_filename = urllib.parse.urljoin(path, each.replace('{FULL}', filename).lstrip('/')) if isfile: new_filename = new_filename.replace('{NAME}', name).replace('{EXT}', extension) else: @@ -253,7 +259,7 @@ def _splitFilename(filename): def generateBlastDict(): ''' - @description: 生成纯暴力字典，支持断点续生成 + @description: Generate pure brute force dictionary, supports resume generation @param {type} @return: ''' @@ -274,13 +280,13 @@ def generateBlastDict(): def generateLengthDict(length): ''' - @description: 生成length长度的字典 + @description: Generate dictionary of specified length @param {type} @return: ''' lst = [0] * length if len(conf.blast_mode_resume_charset) == length and conf.blast_mode_resume_charset != '': - #enumerate()用于将一个可遍历的数据对象(如列表、元组或字符串)组合为一个索引序列 + #enumerate() is used to combine a traversable data object (such as list, tuple or string) into an index sequence for i, letter in enumerate(conf.blast_mode_resume_charset): if conf.blast_mode_custom_charset.find(letter) == -1: outputscreen.error('[+] Invalid resume string: "%s"\n\n' % conf.blast_mode_resume_charset) @@ -309,12 +315,12 @@ def generateLengthDict(length): def generateSingleFuzzDict(path): ''' - @description: 单字典。生成fuzz字典 + @description: Single dictionary. Generate fuzz dictionary @param {type} @return: ''' fuzz_path = urllib.parse.urlparse(conf.url).path - #替换label进行fuzz字典生成 + # Replace label to generate fuzz dictionary if conf.fuzz_mode_label in fuzz_path: for i in loadSingleDict(path): payloads.fuzz_mode_dict.append(fuzz_path.replace(conf.fuzz_mode_label,i)) @@ -324,12 +330,12 @@ def generateSingleFuzzDict(path): sys.exit(1) def generateMultFuzzDict(path): ''' - @description: 多字典。生成fuzz字典 + @description: Multiple dictionaries. Generate fuzz dictionary @param {type} @return: ''' fuzz_path = urllib.parse.urlparse(conf.url).path - #替换label进行fuzz字典生成 + # Replace label to generate fuzz dictionary if conf.fuzz_mode_label in fuzz_path: for i in loadMultDict(path): payloads.fuzz_mode_dict.append(fuzz_path.replace(conf.fuzz_mode_label,i)) @@ -340,7 +346,7 @@ def generateMultFuzzDict(path): def scanModeHandler(): ''' - @description: 扫描模式处理，加载payloads + @description: Handle scanning modes, load payloads @param {type} @return: ''' @@ -351,14 +357,14 @@ def scanModeHandler(): msg = '[*] Use recursive scan: No' outputscreen.warning('\r'+msg+' '*(th.console_width-len(msg)+1)) payloadlists=[] - # fuzz模式处理，只能单独加载 + # Handle fuzz mode, can only load separately if conf.fuzz_mode: outputscreen.warning('[*] Use fuzz mode') if conf.fuzz_mode == 1: return generateSingleFuzzDict(conf.fuzz_mode_load_single_dict) if conf.fuzz_mode == 2: return generateMultFuzzDict(conf.fuzz_mode_load_mult_dict) - # 其他模式处理，可同时加载 + # Handle other modes, can load simultaneously else: if conf.dict_mode: outputscreen.warning('[*] Use dict mode') @@ -375,44 +381,42 @@ def scanModeHandler(): outputscreen.warning('[*] Use paylaod min length: {}'.format(conf.blast_mode_min)) outputscreen.warning('[*] Use paylaod max length: {}'.format(conf.blast_mode_max)) payloadlists.extend(generateBlastDict()) - #TODO:递归爬取url + #TODO: recursively crawl urls if conf.crawl_mode: outputscreen.warning('[*] Use crawl mode') - #自定义header + # Custom header headers = {} if conf.request_headers: try: for header in conf.request_headers.split(','): - k, v = header.split('=') - #print(k,v) - headers[k] = v + key, value = header.split('=') + headers[key] = value except Exception as e: outputscreen.error("[x] Check personalized headers format: header=value,header=value.\n[x] error:{}".format(e)) - # sys.exit() - #自定义ua + # Custom UA if conf.request_header_ua: headers['User-Agent'] = conf.request_header_ua - #自定义cookie + # Custom cookie if conf.request_header_cookie: headers['Cookie'] = conf.request_header_cookie try: response = requests.get(conf.url, headers=headers, timeout=conf.request_timeout, verify=False, allow_redirects=conf.redirection_302, proxies=conf.proxy_server) - #获取页面url + # Get page url if (response.status_code in conf.response_status_code) and response.text: html = etree.HTML(response.text) - #加载自定义xpath用于解析html + # Load custom xpath for parsing html urls = html.xpath(conf.crawl_mode_parse_html) for url in urls: - #去除相似url + # Remove similar urls if urlSimilarCheck(url): - #判断:1.是否同域名 2.netloc是否为空(值空时为同域)。若满足1或2，则添加到temp payload + # Check: 1. Same domain 2. Empty netloc (empty means same domain). Add to temp payload if 1 or 2 is met if (urllib.parse.urlparse(url).netloc == urllib.parse.urlparse(conf.url).netloc) or urllib.parse.urlparse(url).netloc == '': payloads.crawl_mode_dynamic_fuzz_temp_dict.add(url) payloads.crawl_mode_dynamic_fuzz_temp_dict = payloads.crawl_mode_dynamic_fuzz_temp_dict - {'#', ''} if conf.crawl_mode_dynamic_fuzz: - #加载动态fuzz后缀，TODO:独立动态生成字典模块 + # Load dynamic fuzz suffix, TODO: separate dynamic dictionary generation module loadSuffix(os.path.join(paths.DATA_PATH,conf.crawl_mode_dynamic_fuzz_suffix)) - #生成新爬虫动态字典 + # Generate new crawler dynamic dictionary for i in payloads.crawl_mode_dynamic_fuzz_temp_dict: payloads.crawl_mode_dynamic_fuzz_dict.extend(generateCrawlDict(i)) for i in payloads.crawl_mode_dynamic_fuzz_temp_dict: @@ -424,7 +428,6 @@ def scanModeHandler(): payloadlists.extend(set(payloads.crawl_mode_dynamic_fuzz_dict)) except (requests.exceptions.ConnectionError, requests.exceptions.Timeout, requests.exceptions.ReadTimeout) as e: outputscreen.error("[x] Crawler network connection error!plz check whether the target is accessible, error info:{}".format(e)) - # sys.exit() if payloadlists: return payloadlists @@ -434,25 +437,25 @@ def scanModeHandler(): def responseHandler(response): ''' - @description: 处理响应结果 + @description: Handle response results @param {type} @return: ''' - #结果处理阶段 + # Result processing stage try: size = intToSize(int(response.headers['content-length'])) except (KeyError, ValueError): size = intToSize(len(response.content)) - #跳过大小为skip_size的页面 + # Skip pages with size equal to skip_size if size == conf.skip_size: return - #自动识别404-判断是否与获取404页面特征匹配 + # Auto detect 404 - check if matches 404 page characteristics if conf.auto_check_404_page: if hashlib.md5(response.content).hexdigest() in conf.autodiscriminator_md5: return - #自定义状态码显示 + # Custom status code display if response.status_code in conf.response_status_code: msg = '[{}]'.format(str(response.status_code)) if conf.response_header_content_type: @@ -468,15 +471,15 @@ def responseHandler(response): outputscreen.error('\r'+msg+' '*(th.console_width-len(msg)+1)) else: outputscreen.info('\r'+msg+' '*(th.console_width-len(msg)+1)) - #已去重复，结果保存。NOTE:此处使用response.url进行文件名构造，解决使用-iL参数时，不能按照域名来命名文件名的问题 - #使用replace()，替换`:`，修复window下不能创建有`:`的文件问题 + # Already deduplicated, save results. NOTE: Using response.url to construct filename here solves the issue of not being able to name files by domain when using -iL parameter + # Use replace() to replace `:` to fix the issue of not being able to create files with `:` on Windows saveResults(urllib.parse.urlparse(response.url).netloc.replace(':','_'),msg) - #关于递归扫描。响应在自定义状态码中时，添加判断是否进行递归扫描 + # About recursive scanning. When response is in custom status codes, add check for recursive scanning if response.status_code in conf.recursive_status_code: if conf.recursive_scan: recursiveScan(response.url,payloads.all_payloads) - #自定义正则匹配响应 + # Custom regex response matching if conf.custom_response_page: pattern = re.compile(conf.custom_response_page) if pattern.search(response.text): @@ -484,39 +487,38 @@ def responseHandler(response): def worker(): ''' - @description: 封包发包穷举器 + @description: Packet sending enumerator @param {type} @return: ''' payloads.current_payload = tasks.all_task.get() - #1自定义封包阶段 - #自定义header + #1 Custom packet phase + # Custom header headers = {} if conf.request_headers: try: for header in conf.request_headers.split(','): - k, v = header.split('=') - #print(k,v) - headers[k] = v + key, value = header.split('=') + headers[key] = value except Exception as e: outputscreen.error("[x] Check personalized headers format: header=value,header=value.\n[x] error:{}".format(e)) sys.exit() - #自定义ua + # Custom UA if conf.request_header_ua: headers['User-Agent'] = conf.request_header_ua - #自定义cookie + # Custom cookie if conf.request_header_cookie: headers['Cookie'] = conf.request_header_cookie try: - #2进入发送请求流程 - #延迟请求 + #2 Enter request sending flow + # Delay request if conf.request_delay: random_sleep_second = random.randint(0,abs(conf.request_delay)) time.sleep(random_sleep_second) response = requests.request(conf.request_method, payloads.current_payload, headers=headers, timeout=conf.request_timeout, verify=False, allow_redirects=conf.redirection_302, proxies=conf.proxy_server) - #3进入结果处理流程 + #3 Enter result processing flow responseHandler(response) except requests.exceptions.Timeout as e: #outputscreen.error('[x] timeout! url:{}'.format(payloads.current_payload)) @@ -525,42 +527,40 @@ def worker(): # outputscreen.error('[x] error:{}'.format(e)) pass finally: - #更新进度条 + # Update progress bar tasks.task_count += 1 bar.log.update(tasks.task_count) -def boss(): - ''' - @description: worker控制器 - @param {type} - @return: - ''' +def task_dispatcher(): + """ + Worker controller that dispatches tasks to workers. + """ while not tasks.all_task.empty(): worker() def bruter(url): ''' - @description: 扫描插件入口函数 - @param {url:目标} + @description: Scanning plugin entry function + @param {url: target} @return: ''' - #url初始化 + # URL initialization conf.parsed_url = urllib.parse.urlparse(url) - #填补协议 + # Add protocol if conf.parsed_url.scheme != 'http' and conf.parsed_url.scheme != 'https': url = 'http://' + url conf.parsed_url = urllib.parse.urlparse(url) - #全局target的url，给crawl、fuzz模块使用。XXX:要放在填补url之前，否则fuzz模式会出现这样的问题：https://target.com/phpinfo.{dir}/ + # Global target url for crawl and fuzz modules. XXX: Must be placed before URL padding, otherwise fuzz mode will have issues like: https://target.com/phpinfo.{dir}/ conf.url = url - #填补url后的/ + # Add trailing / to URL if not url.endswith('/'): url = url + '/' - #打印当前target + # Print current target msg = '[+] Current target: {}'.format(url) outputscreen.success('\r'+msg+' '*(th.console_width-len(msg)+1)) - #自动识别404-预先获取404页面特征 + # Auto detect 404 - pre-get 404 page characteristics if conf.auto_check_404_page: outputscreen.warning("[*] Launching auto check 404") # Autodiscriminator (probably deprecated by future diagnostic subsystem) @@ -569,37 +569,39 @@ def bruter(url): if notfound_type == Inspector.TEST404_MD5 or notfound_type == Inspector.TEST404_OK: conf.autodiscriminator_md5.add(result) - #加载payloads + # Load payloads payloads.all_payloads = scanModeHandler() - #FIXME:设置后缀名。当前以拼接方式实现，遍历一遍payload。 + #FIXME: Set file extensions. Currently implemented by concatenation, traverse all payloads. try: if conf.file_extension: outputscreen.warning('[+] Use file extentsion: {}'.format(conf.file_extension)) - for i in range(len(payloads.all_payloads)): - payloads.all_payloads[i] += conf.file_extension - except: - outputscreen.error('[+] plz check extension!') - sys.exit() - #debug模式，打印所有payload，并退出 + for idx in range(len(payloads.all_payloads)): + payloads.all_payloads[idx] += conf.file_extension + except (AttributeError, TypeError) as e: + outputscreen.error(f'[+] Please check extension configuration: {e}') + sys.exit(1) + # Debug mode, print all payloads and exit if conf.debug: outputscreen.blue('[+] all payloads:{}'.format(payloads.all_payloads)) sys.exit() - #payload入队task队列 + # Enqueue payloads to task queue for payload in payloads.all_payloads: - #FIXME:添加fuzz模式时，引入的url_payload构造判断 + #FIXME: URL payload construction check added when fuzz mode was introduced if conf.fuzz_mode: - url_payload = conf.parsed_url.scheme + '://' + conf.parsed_url.netloc + payload + # Use urljoin to correctly handle URL concatenation, avoiding double slash issue + base_url = conf.parsed_url.scheme + '://' + conf.parsed_url.netloc + '/' + url_payload = urllib.parse.urljoin(base_url, payload.lstrip('/')) else: - url_payload = url + payload - #print(url_payload) - #payload入队，等待处理 + # Use urljoin to correctly handle URL concatenation, avoiding double slash issue + url_payload = urllib.parse.urljoin(url, payload.lstrip('/')) + # Enqueue payload, waiting for processing tasks.all_task.put(url_payload) - #设置进度条长度，若是递归模式或爬虫模式，则不设置任务队列长度，即无法显示进度，仅显示耗时 + # Set progress bar length. If recursive mode or crawler mode, do not set task queue length, i.e., cannot show progress, only show time if not conf.recursive_scan: - #NOTE:这里取所有payloads的长度*target数量计算任务总数，修复issue#2 + #NOTE: Take length of all payloads * number of targets to calculate total tasks, fix issue#2 tasks.task_length = len(payloads.all_payloads)*conf.target_nums bar.log.start(tasks.task_length) - #FIXME:循环任务数不能一次性取完所有的task，暂时采用每次执行30个任务。这样写还能解决hub.LoopExit的bug + #FIXME: Cannot take all tasks at once in loop, temporarily execute 30 tasks each time. This also solves hub.LoopExit bug while not tasks.all_task.empty(): - all_task = [gevent.spawn(boss) for i in range(conf.request_limit)] + all_task = [gevent.spawn(task_dispatcher) for _ in range(conf.request_limit)] gevent.joinall(all_task) diff --git a/lib/controller/engine.py b/lib/controller/engine.py index 90fe6d3..3598147 100644 --- a/lib/controller/engine.py +++ b/lib/controller/engine.py @@ -1,13 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-05-01 20:19:40 -''' - import gevent import sys import time @@ -23,27 +16,27 @@ def initEngine(): th.result = [] th.thread_num = conf.thread_num th.target = conf.target - #是否继续扫描标志位 + # Whether to continue scanning flag th.is_continue = True - #控制台宽度 + # Console width th.console_width = getTerminalSize()[0] - 2 - #记录开始时间 + # Record start time th.start_time = time.time() msg = '[+] Set the number of thread: %d' % th.thread_num outputscreen.success(msg) def scan(): while True: - #协程模式 + # Coroutine mode if th.target.qsize() > 0 and th.is_continue: target = str(th.target.get(timeout=1.0)) else: break try: - #对每个target进行检测 + # Perform detection on each target bruter(target) except Exception: - #抛出异常时，添加errmsg键值 + # When exception is thrown, add errmsg key-value th.errmsg = traceback.format_exc() th.is_continue = False diff --git a/lib/core/__init__.py b/lib/core/__init__.py index a7e8013..7847780 100644 --- a/lib/core/__init__.py +++ b/lib/core/__init__.py @@ -1,9 +1,3 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 16:11:27 -@LastEditTime: 2019-04-11 20:11:24 -''' diff --git a/lib/core/common.py b/lib/core/common.py index 9797bb4..d7340a0 100644 --- a/lib/core/common.py +++ b/lib/core/common.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: ttttmr -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-05-29 16:49:22 -''' import os.path import sys @@ -16,13 +10,12 @@ from lib.core.data import cmdLineOptions, conf, paths, payloads from lib.core.enums import COLOR -from lib.core.setting import BANNER from thirdlib.colorama import Back, Fore, Style, init init(autoreset=True) class Outputscreen: """ - 显示颜色类 + Display color class """ def info(self, s): print(Style.BRIGHT+Fore.WHITE + str(s) + Fore.RESET+Style.RESET_ALL) @@ -40,14 +33,14 @@ def error(self, s): def blue(self, s): print(Style.BRIGHT+Fore.BLUE + str(s) + Fore.RESET+Style.RESET_ALL) -#创建outputscreen对象，用于输出各种颜色的信息 +# Create outputscreen object for outputting various colored information outputscreen=Outputscreen() def setPaths(): """ - 设置全局绝对路径 + Set global absolute paths """ - # 根目录 + # Root directory root_path = paths.ROOT_PATH # datapath paths.DATA_PATH = os.path.join(root_path, "data") @@ -72,15 +65,8 @@ def setPaths(): #print(root_path,paths.DATA_PATH,paths.SCRIPT_PATH,paths.OUTPUT_PATH,paths.CONFIG_PATH) #print(paths.WEAK_PASS,paths.LARGE_WEAK_PASS,paths.UA_LIST_PATH) -def banner(): - ''' - @description: 打印banner - @param {type} - @return: - ''' - outputscreen.blue(BANNER) -# 将'192.168.1.1-192.168.1.100'分解成ip地址列表 +# Decompose '192.168.1.1-192.168.1.100' into IP address list def genIP(ip_range): ''' print (genIP('192.18.1.1-192.168.1.3')) @@ -97,45 +83,45 @@ def ip2num(ip): start ,end = [ip2num(x) for x in ip_range.split('-')] return [num2ip(num) for num in range(start,end+1) if num & 0xff] -# 识别目标，转换成列表形式 +# Identify targets, convert to list form def parseTarget(target): - lists=[] - ipv4withmask_re=re.compile("^(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])/(3[0-2]|[1-2]?[0-9])$") - ipv4range_re=re.compile("^(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])-(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$") + targets=[] + ipv4withmask_re = re.compile(r"^(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])/(3[0-2]|[1-2]?[0-9])$") + ipv4range_re = re.compile(r"^(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])-(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])$") try: - # 尝试解析url + # Try to parse url parsed_url=urllib.parse.urlparse(target) - # 判断不带http + # Check if not http if parsed_url.scheme != 'http': - # 判断IP/Mask格式 + # Check IP/Mask format if ipv4withmask_re.search(parsed_url.path): - # 是子网还是网址 e.g. 192.168.1.1/24 or http://192.168.1.1/24 + # Is it subnet or website e.g. 192.168.1.1/24 or http://192.168.1.1/24 infomsg = "[*] %s is IP/Mask[Y] or URL(http://)[n]? [Y/n]" %target outputscreen.info(infomsg) flag =input() if flag in ('N', 'n', 'no', 'No', 'NO'): - # 按链接处理 e.g. http://192.168.1.1/24 - lists.append(target) + # Process as link e.g. http://192.168.1.1/24 + targets.append(target) else: - # 按子网处理 e.g. 192.168.1.1/24 - lists=list(ipaddress.ip_interface(target).network) - # 判断网络范围格式 e.g. 192.168.1.1-192.168.1.100 + # Process as subnet e.g. 192.168.1.1/24 + targets=list(ipaddress.ip_interface(target).network) + # Check network range format e.g. 192.168.1.1-192.168.1.100 elif ipv4range_re.search(target): - lists=genIP(target) - # 按照链接处理 + targets=genIP(target) + # Process as link else: - lists.append(target) - # 为http://格式 + targets.append(target) + # For http:// format else: - lists.append(target) + targets.append(target) except: - # 识别失败 + # Identification failed pass - return lists + return targets def intToSize(bytes): ''' - @description: bits大小转换，对人类友好 + @description: bits size conversion, human friendly @param {type} @return: ''' @@ -149,9 +135,9 @@ def intToSize(bytes): def urlSimilarCheck(url): ''' - @description: url相似度分析，当url路径和参数键值类似时，则判为重复，参考某人爬虫 + @description: url similarity analysis, when url path and parameter key values are similar, it is judged as duplicate, referencing someone's crawler @param {type} - @return: 非重复返回True + @return: Return True if not duplicate ''' url_struct = urllib.parse.urlparse(url) query_key = '|'.join(sorted([i.split('=')[0] for i in url_struct.query.split('&')])) diff --git a/lib/core/data.py b/lib/core/data.py index 9b81c6c..e4f9902 100644 --- a/lib/core/data.py +++ b/lib/core/data.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-04-10 17:46:40 -''' from lib.core.datatype import AttribDict @@ -23,11 +17,11 @@ # object to control engine th = AttribDict() -#创建payloads字典对象存储payloads +# Create payloads dictionary object to store payloads payloads = AttribDict() -#创建tasks字典对象存储tasks +# Create tasks dictionary object to store tasks tasks = AttribDict() -#创建进度条对象存储进度 +# Create progress bar object to store progress bar = AttribDict() \ No newline at end of file diff --git a/lib/core/datatype.py b/lib/core/datatype.py index 7eed042..d7e093d 100644 --- a/lib/core/datatype.py +++ b/lib/core/datatype.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-04-10 17:48:54 -''' import copy import types diff --git a/lib/core/enums.py b/lib/core/enums.py index 7b55cd2..18bd67e 100644 --- a/lib/core/enums.py +++ b/lib/core/enums.py @@ -1,22 +1,16 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-04-14 10:36:27 -''' class COLOR: - black = 30 # 黑色 - red = 31 # 红色 - green = 32 # 绿色 - yellow = 33 # 黄色 - blue = 34 # 蓝色 - purple = 35 # 紫红色 - cyan = 36 # 青蓝色 - white = 37 # 白色 + black = 30 # black + red = 31 # red + green = 32 # green + yellow = 33 # yellow + blue = 34 # blue + purple = 35 # purple/red + cyan = 36 # cyan/blue + white = 37 # white class BRUTER_RESULT_STATUS: FAIL = 0 diff --git a/lib/core/option.py b/lib/core/option.py index 1a9e379..d5f12ce 100644 --- a/lib/core/option.py +++ b/lib/core/option.py @@ -1,24 +1,15 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: ttttmr -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-05-29 16:52:42 -''' - -import imp + import os -import queue import sys -import time -import ipaddress +from gevent.queue import Queue from lib.controller.bruter import loadConf from lib.core.common import parseTarget, outputscreen -from lib.core.data import conf, paths -from thirdlib.IPy.IPy import IP +from lib.core.data import conf + def initOptions(args): EngineRegister(args) @@ -27,91 +18,89 @@ def initOptions(args): def EngineRegister(args): - """ - 加载并发引擎模块 - """ - conf.engine_mode = 'coroutine' - - #设置线程数 - if args.thread_num > 200 or args.thread_num < 1: - msg = '[*] Invalid input in [-t](range: 1 to 200), has changed to default(30)' - outputscreen.warning(msg) + conf.engine_mode = "coroutine" + + if not isinstance(args.thread_num, int) or args.thread_num < 1 or args.thread_num > 200: + outputscreen.warning("[*] Invalid input in [-t] (range: 1..200). Using default: 30") conf.thread_num = 30 - return - conf.thread_num = args.thread_num + else: + conf.thread_num = args.thread_num + def BruterRegister(args): - """ - 配置bruter模块 - """ + # gán debug flag (bool) + conf.debug = bool(getattr(args, "debug", False)) - if args.load_config_file: - #加载配置文件 + if getattr(args, "load_config_file", False): loadConf() else: - outputscreen.error("[+] Function development, coming soon!please use -lcf parameter") - if args.debug: - conf.debug = args.debug - else: - conf.debug = args.debug - sys.exit() + outputscreen.error("[+] Feature not ready. Please use -lcf/--load-config-file to specify a config file.") + sys.exit(1) + def TargetRegister(args): - """ - 加载目标模块 - """ - msg = '[*] Initialize targets...' - outputscreen.warning(msg) - - #初始化目标队列 - conf.target = queue.Queue() - - # 用户输入入队 - if args.target_input: - # 尝试解析目标地址 + outputscreen.warning("[*] Initialize targets...") + + # Initialize target queue + conf.target = Queue() + + target_input = getattr(args, "target_input", None) + if target_input: try: - lists=parseTarget(args.target_input) - except: - helpmsg = "Invalid input in [-i], Example: -i [http://]target.com or 192.168.1.1[/24] or 192.168.1.1-192.168.1.100" - outputscreen.error(helpmsg) - sys.exit() - # 判断处理量 - if (len(lists))>100000: - warnmsg = "[*] Loading %d targets, Maybe it's too much, continue? [y/N]" % (len(lists)) - outputscreen.warning(warnmsg) - flag =input() - if flag in ('Y', 'y', 'yes', 'YES','Yes'): - pass - else: - msg = '[-] User quit!' - outputscreen.warning(msg) - sys.exit() - msg = '[+] Load targets from: %s' % args.target_input - outputscreen.success(msg) - # save to conf - for target in lists: - conf.target.put(target) + targets = parseTarget(target_input) + except Exception as e: + outputscreen.error( + "Invalid input in [-i]. Example:\n" + " -i [http://]target.com\n" + " -i 192.168.1.1[/24]\n" + " -i 192.168.1.1-192.168.1.100" + ) + if conf.debug: + outputscreen.error(f"[debug] parseTarget error: {e!r}") + sys.exit(1) + + if len(targets) > 100_000: + outputscreen.warning(f"[*] Loading {len(targets)} targets. Maybe it's too much, continue? [y/N]") + try: + ans = input().strip() + except EOFError: + ans = "" + if ans not in ("Y", "y", "yes", "YES", "Yes"): + outputscreen.warning("[-] User quit!") + sys.exit(1) + + outputscreen.success(f"[+] Load targets from: {target_input}") + for t in targets: + conf.target.put(t) conf.target_nums = conf.target.qsize() - # 文件读入入队 - elif args.target_file: - if not os.path.isfile(args.target_file): - msg = '[-] TargetFile not found: %s' % args.target_file - outputscreen.error(msg) - sys.exit() - msg = '[+] Load targets from: %s' % args.target_file - outputscreen.success(msg) - with open(args.target_file, 'r', encoding='utf-8') as f: - targets = f.readlines() - for target in targets: - target=target.strip('\n') - parsed_target=parseTarget(target) - for i in parsed_target: - conf.target.put(i) + elif getattr(args, "target_file", None): + target_file = args.target_file + if not os.path.isfile(target_file): + outputscreen.error(f"[-] Target file not found: {target_file}") + sys.exit(1) + + outputscreen.success(f"[+] Load targets from: {target_file}") + try: + with open(target_file, "r", encoding="utf-8") as f: + for line in f: + line = line.strip() + if not line: + continue + try: + parsed = parseTarget(line) + except Exception as e: + if conf.debug: + outputscreen.error(f"[debug] parseTarget failed for line '{line}': {e!r}") + continue + for item in parsed: + conf.target.put(item) + except Exception as e: + outputscreen.error(f"[-] Failed to read target file: {e}") + sys.exit(1) + conf.target_nums = conf.target.qsize() - #验证目标数量 if conf.target.qsize() == 0: - errormsg = msg = '[!] No targets found.Please load targets with [-i|-iF]' - outputscreen.error(errormsg) - sys.exit() \ No newline at end of file + outputscreen.error("[!] No targets found. Please load targets with [-i | -iF]") + sys.exit(1) diff --git a/lib/core/setting.py b/lib/core/setting.py index 1032920..7b9de4b 100644 --- a/lib/core/setting.py +++ b/lib/core/setting.py @@ -1,18 +1,4 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-04-11 15:58:40 -''' -VERSION = "1.1" -BANNER = f""" - ##### # ##### # # ## ##### - # # # # # ## ## # # # # - # # # # # # ## # # # # # - # # # ##### # # ###### ##### - # # # # # # # # # # - ##### # # # # # # # # v{VERSION} -""" \ No newline at end of file +VERSION = "1.1" \ No newline at end of file diff --git a/lib/parse/__init__.py b/lib/parse/__init__.py index 83060aa..7847780 100644 --- a/lib/parse/__init__.py +++ b/lib/parse/__init__.py @@ -1,9 +1,3 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 20:11:49 -@LastEditTime: 2019-04-11 20:11:53 -''' diff --git a/lib/parse/cmdline.py b/lib/parse/cmdline.py index ac9985a..e8a78a4 100644 --- a/lib/parse/cmdline.py +++ b/lib/parse/cmdline.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: ttttmr -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-07-22 21:46:31 -''' import sys import argparse diff --git a/lib/plugins/__init__.py b/lib/plugins/__init__.py index 31ecb97..7847780 100644 --- a/lib/plugins/__init__.py +++ b/lib/plugins/__init__.py @@ -1,9 +1,3 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 20:11:08 -@LastEditTime: 2019-04-11 20:11:12 -''' diff --git a/lib/plugins/inspector.py b/lib/plugins/inspector.py index 1bec143..a300b30 100644 --- a/lib/plugins/inspector.py +++ b/lib/plugins/inspector.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: ttttmr -@Date: 2019-05-01 12:07:54 -@LastEditTime: 2019-06-26 00:25:14 -''' import hashlib import random @@ -44,7 +38,8 @@ def _give_it_a_try(self): random.seed() s.append(chr(random.randrange(97, 122))) s = "".join(s) - target = self.target + s + # Use urljoin to correctly handle URL concatenation, avoiding double slash issue + target = urllib.parse.urljoin(self.target, s) outputscreen.success("[+] Checking with: {}".format(target)) diff --git a/lib/utils/__init__.py b/lib/utils/__init__.py index 6eabc61..7847780 100644 --- a/lib/utils/__init__.py +++ b/lib/utils/__init__.py @@ -1,9 +1,3 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-11 16:11:27 -@LastEditTime: 2019-04-11 20:04:19 -''' diff --git a/lib/utils/config.py b/lib/utils/config.py index fe83375..f9db1e3 100644 --- a/lib/utils/config.py +++ b/lib/utils/config.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: ttttmr -@Date: 2019-04-11 09:49:16 -@LastEditTime: 2019-05-29 16:49:43 -''' from configparser import ConfigParser from lib.core.data import paths @@ -20,8 +14,8 @@ def _get_option(section, option): cf = ConfigParser() cf.read(paths.CONFIG_PATH) return cf.get(section=section, option=option) - except: - outputscreen.warning('Missing essential options, please check your config-file.') + except Exception as e: + outputscreen.warning(f'Missing essential options, please check your config-file: {e}') return '' diff --git a/lib/utils/console.py b/lib/utils/console.py index 2b7a25f..6f21e77 100644 --- a/lib/utils/console.py +++ b/lib/utils/console.py @@ -1,12 +1,6 @@ #!/usr/bin/env python3 # -*- coding: utf-8 -*- -''' -@Author: xxlin -@LastEditors: xxlin -@Date: 2019-04-10 13:27:58 -@LastEditTime: 2019-04-11 15:57:17 -''' """ getTerminalSize() @@ -95,8 +89,10 @@ def ioctl_GWINSZ(fd): pass if not cr: try: - cr = (env.get('LINES'), env.get('COLUMNS')) + cr = (os.environ.get('LINES'), os.environ.get('COLUMNS')) except Exception: return None + if cr[0] is None or cr[1] is None: + return None return int(cr[1]), int(cr[0]) diff --git a/requirement.txt b/requirement.txt index efc2fd7..3990adc 100644 --- a/requirement.txt +++ b/requirement.txt @@ -1,4 +1,4 @@ -gevent==20.12.1 -requests==2.31.0 -progressbar2==3.53.1 -lxml==4.5.0 \ No newline at end of file +gevent +requests +progressbar2 +lxml \ No newline at end of file