You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Dedoc is an open universal system for converting documents to a unified output format. It extracts a document’s logical structure and content, its tables, text formatting and metadata. The document’s content is represented as a tree storing headings and lists of any level. Dedoc can be integrated in a document contents and structure analysis system as a separate module.
7
+
Dedoc is an open universal system for converting documents to a unified output format.
8
+
It extracts a document’s logical structure and content, its tables, text formatting and metadata.
9
+
The document’s content is represented as a tree storing headings and lists of any level.
10
+
Dedoc can be integrated in a document contents and structure analysis system as a separate module.
8
11
9
12
## Features and advantages
10
-
Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats. Document structure extraction is fully automatic regardless of input data type. Metadata and text formatting is also extracted automatically.
13
+
Dedoc is implemented in Python and works with semi-structured data formats (DOC/DOCX, ODT, XLS/XLSX, CSV, TXT, JSON) and none-structured data formats like images (PNG, JPG etc.), archives (ZIP, RAR etc.), PDF and HTML formats.
14
+
Document structure extraction is fully automatic regardless of input data type.
15
+
Metadata and text formatting are also extracted automatically.
11
16
12
17
In 2022, the system won a grant to support the development of promising AI projects from the [Innovation Assistance Foundation (Фонд содействия инновациям)](https://fasie.ru/).
13
18
@@ -16,45 +21,189 @@ In 2022, the system won a grant to support the development of promising AI proje
16
21
* Support for extracting document structure out of nested documents having different formats.
17
22
* Extracting various text formatting features (indentation, font type, size, style etc.).
18
23
* Working with documents of various origin (statements of work, legal documents, technical reports, scientific papers) allowing flexible tuning for new domains.
19
-
* Working with PDF documents containinng a text layer:
20
-
* Support to automatically determine the correctness of the text layer in PDF documents;
21
-
* Extract containing and formatting from PDF-documents with a text layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification.
22
-
Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats:
24
+
* Working with PDF documents containing a textual layer:
25
+
* Support to automatically determine the correctness of the textual layer in PDF documents;
26
+
* Extract containing and formatting from PDF-documents with a textual layer using the developed interpreter of the virtual stack machine for printing graphics according to the format specification.
27
+
*Extracting table data from DOC/DOCX, PDF, HTML, CSV and image formats:
23
28
* Recognizing a physical structure and a cell text for complex multipage tables having explicit borders with the help of contour analysis.
24
29
* Working with scanned documents (image formats and PDF without text layer):
25
30
* Using Tesseract, an actively developed OCR engine from Google, together with image preprocessing methods.
26
31
* Utilizing modern machine learning approaches for detecting a document orientation, detecting single/multicolumn document page, detecting bold text and extracting hierarchical structure based on the classification of features extracted from document images.
27
32
33
+
## Impact
34
+
This project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part).
35
+
Dedoc is in demand for information analytic systems, information leak monitoring systems, as well as for natural language processing systems.
36
+
The library is intended for application use by developers of systems for automatic analysis and structuring of electronic documents, including for further search in electronic documents.
28
37
29
-
This project may be useful as a first step of automatic document analysis pipeline (e.g. before the NLP part)
38
+
# Online-Documentation
39
+
Relevant documentation of the dedoc is available [here](https://dedoc.readthedocs.io/en/latest/)
30
40
31
-
This project has REST Api and you can run it in Docker container
32
-
To read full Dedoc documentation run the project and go to localhost:1231.
33
-
41
+
# Installation instructions
42
+
****************************************
43
+
This project has REST Api and you can run it in Docker container.
44
+
Also, dedoc can be installed as a library via `pip`.
45
+
There are two ways to install and run dedoc as a web application or a library that are described below.
34
46
35
-
## Run the project
36
-
How to build and run the project
37
47
38
-
Ensure you have Git and Docker installed
39
-
40
-
Clone the project
48
+
## Install and run dedoc using docker
49
+
50
+
You should have [`git`] (https://git-scm.com) and [`docker`](https://www.docker.com) installed for running dedoc by this method.
51
+
This method is more flexible because it doesn't depend on the operating system and other user's limitations,
52
+
still, the docker application should be installed and configured properly.
53
+
54
+
If you don't need to change the application configuration, you may use the built docker image as well.
55
+
56
+
### 1. Pull the image
41
57
```bash
42
-
git clone https://github.com/ispras/dedoc.git
58
+
docker pull dedocproject/dedoc
59
+
```
43
60
44
-
cd dedoc/
61
+
### 2. Run the container
62
+
```bash
63
+
docker run -p 1231:1231 --rm dedocproject/dedoc python3 /dedoc_root/dedoc/main.py
45
64
```
46
-
47
-
Ensure you have Docker installed.
48
-
Start 'Dedoc' on the port 1231:
49
-
```bash
65
+
66
+
Go to [dockerhub](https://hub.docker.com/r/dedocproject/dedoc) to get more information about available dedoc images.
67
+
68
+
If you need to change some application settings, you may update `config.py` according to your needs and re-build the image.
69
+
You can build and run image:
70
+
71
+
### 1. Clone the repository
72
+
```bash
73
+
git clone https://github.com/ispras/dedoc
74
+
```
75
+
76
+
### 2. Go to the `dedoc` directory
77
+
```bash
78
+
cd dedoc
79
+
```
80
+
81
+
### 3. Build the image and run the application
82
+
```bash
50
83
docker-compose up --build
51
84
```
52
85
53
-
Start Dedoc with tests:
54
-
```bash
86
+
### 4. Run container with tests
87
+
```bash
55
88
test="true" docker-compose up --build
56
-
```
89
+
```
90
+
91
+
If you need to change some application settings, you may update `config.py` according to your needs and re-build the image.
92
+
93
+
94
+
## Install dedoc using pip
95
+
96
+
If you don't want to use docker for running the application, it's possible to run dedoc locally.
97
+
However, it isn't suitable for any operating system (`Ubuntu 20+` is recommended) and
98
+
there may be not enough machine's resources for its work.
99
+
You should have `python` (`python3.8`, `python3.9` are recommended) and `pip` installed.
Now you can go to the `localhost:1231` and look at the docs and examples.
184
+
185
+
## Option: You can change the port of service:
186
+
you need to change environment DOCREADER_PORT
187
+
188
+
1. For local service launching on your_port (1166 example). [Install instruction from sources](#Install-and-run-dedoc-from-sources) and launch with environment:
You can install a trusted library `torch` (as a verified version of the library, verified by tools developed by the Ivannikov Institute for System Programming of the Russian Academy of Sciences).
166
+
167
+
First you need to install two required packages.:
168
+
169
+
.. code-block:: bash
170
+
171
+
sudo apt-get install -y mpich intel-mkl
172
+
173
+
Second you need to install torch and torchvision from built wheels:
0 commit comments