Changes for version 0.0.6

Documentated in files changes.rst.

Author: JorjMcKie

docs/src/api.rst

@@ -10,25 +10,39 @@ API
 
     Prints the version of the library.
 
-.. method:: to_markdown(doc: pymupdf.Document | str, *, pages: list | range | None = None, hdr_info: Any = None, write_images: bool = False, margins=(0, 50, 0, 50), page_chunks: bool = False) -> str | list[dict]
+.. method:: to_markdown(doc: pymupdf.Document | str, *, pages: list | range | None = None, hdr_info: Any = None, write_images: bool = False, dpi: int = 150, margins=(0, 50, 0, 50), page_chunks: bool = False, page_width: float = 612, page_height: float = None, table_strategy="lines_strict", graphics_limit: int = None) -> str | list[dict]
 
-    Read the pages of the file and outputs the text of its pages in |Markdown| format. How this should happen in detail can be influenced by a number of parameters. Please note that there exists support for building page chunks from the |Markdown|  text.
+    Read the pages of the file and outputs the text of its pages in |Markdown| format. How this should happen in detail can be influenced by a number of parameters. Please note that there exists **support for building page chunks** from the |Markdown|  text.
 
-    :arg Document,str doc: the file, to be specified either as a file path string, or as a |PyMuPDF| Document (created via `pymupdf.open`).
+    :arg Document,str doc: the file, to be specified either as a file path string, or as a |PyMuPDF| Document (created via `pymupdf.open`). In order to use `pathlib.Path` specifications, Python file-like objects, documents in memory etc. you **must** use a |PyMuPDF| Document.
 
-    :arg list,range pages: optional, the pages to consider for output. If omitted all pages are processed.
+    :arg list pages: optional, the pages to consider for output (caution: specify 0-based page numbers). If omitted all pages are processed.
 
-    :arg hdr_info: optional, a callable (or an object having a method named `hdr_info`) which accepts a text span and delivers a string of 0 up to 6 "#" characters which should be used to identify headers in the markdown text. If omitted, a full document scan will be performed to find the most popular font sizes and derive header levels based on this. For instance, to avoid generating any lines tagged as headers specify `hdr_info=lambda s: ""`.
+    :arg hdr_info: optional. Use this if you want to provide your own header detection logic. This may be a callable or an object having a method named `get_header_id`. It must accept a text span (a span dictionary as contained in `extractDict <https://pymupdf.readthedocs.io/en/latest/textpage.html#span-dictionary>`_) and has optional access to the owning `Page <https://pymupdf.readthedocs.io/en/latest/page.html>`_ object. It must return a string "" or up to 6 "#" characters followed by 1 space. If omitted, a full document scan will be performed to find the most popular font sizes and derive header levels based on this. For instance, avoid any headers by specifying `hdr_info=lambda s: ""`.
 
-    :arg bool write_images: when encountering images or vector graphics, PNG images will be generated from the respective page area. Markdown references will be generated pointing to these images. Any text contained in these areas will not be included in the output. Therefore, if your document has text written on full page images, make sure to set this parameter to `False`.
+    :arg bool write_images: when encountering images or vector graphics, PNG images will be created from the respective page area and stored in the folder of the document. Markdown references will be generated pointing to these images. Any text contained in these areas will not be included in the text output (but appear as part of the images). Therefore, if your document has text written on full page images, make sure to set this parameter to `False`.
 
-    :arg float,list margins: a float or a list of up to 4 floats specifying page borders. If 4 floats are provided, they are assumed to be the values left, top, right, bottom, in this sequence. Only content below top and above bottom, etc. will be considered for processing. If a single float value is provided, it will be taken as the value for all 4 border values. A pair of numbers is assumed to specify top and bottom.
+    :arg int dpi: specify the desired image resolution in dots per inch. Relevant only if `write_images=True`. Default value is 150.
+
+    :arg float,list margins: a float or a sequence of 2 or 4 floats specifying page borders. Only objects inside the margins will be considered for output.
+    
+        * `margin=f` yields `(f, f, f, f)` for `(left, top, right, bottom)`.
+        * `(top, bottom)` yields  `(0, top, 0, bottom)`.
+        * To always read full pages, use `margins=0`.
+
+    :arg float page_width: specify a desired page width. This is ignored for documents with a fixed page width like PDF, XPS etc. **Reflowable** documents however, like e-books, office or text files have no fixed page dimensions and by default are assumed to have Letter format width (612) and an **"infinite"** page height. This means that the full document is treated as one large page.
+
+    :arg float page_height: specify a desired page height. For relevance see the `page_width` parameter. If using the default `None`, the document will appear as one large page with a width of `page_width`. Consequently in this case, no markdown page separators will occur (except the final one), respectively only one page chunk will be returned.
 
+    :arg str table_strategy: table detection strategy. Default is `"lines_strict"` which ignores background colors. In some occasions, other strategies may be more successful, for example `"lines"` which uses all vector graphics objects for detection.
+
+    :arg int graphics_limit: use this to limit dealing with excess amounts of vector graphics elements. Typically, scientific documents or pages simulating text using graphics commands may contain tens of thousands of these objects. As vector graphics are used for table detection mainly, analyzing pages of this kind may result in excessive runtimes. You can exclude problematic pages via `graphics_limit=5000`. The respective pages will then be ignored and be represented by one message line in the output text.
+
     :arg bool page_chunks: if `True` the output will be a list of `Document.page_count` dictionaries (one per page). Each dictionary has the following structure:
 
         - **"metadata"** - a dictionary consisting of the document's metadata `Document.metadata <https://pymupdf.readthedocs.io/en/latest/document.html#Document.metadata>`_, enriched with additional keys **"file_path"** (the file name), **"page_count"** (number of pages in document), and **"page_number"** (1-based page number).
 
-        - **"toc_items"** - a list of Table of Contents items pointing to this page. Each item of this list has the format `[lvl, title, pagenumber]`, where `lvl` is the hierachy level, `title` a string and `pagenumber` the 12-based page number.
+        - **"toc_items"** - a list of Table of Contents items pointing to this page. Each item of this list has the format `[lvl, title, pagenumber]`, where `lvl` is the hierachy level, `title` a string and `pagenumber` the 1-based page number.
 
         - **"tables"** - a list of tables on this page. Each item is a dictionary with keys "bbox", "row_count" and "col_count". Key "bbox" is a `pymupdf.Rect` in tuple format of the table's position on the page.
 
@@ -42,7 +56,7 @@ API
 
 .. method:: LlamaMarkdownReader(*args, **kwargs)
 
-    Create a `pdf_markdown_reader.PDFMarkdownReader` using the `LlamaIndex <https://pypi.org/project/llama-index/>`_ package. Please note that this package will **not automatically be installed** when installing **pymupdf4llm**.
+    Create a `pdf_markdown_reader.PDFMarkdownReader` using the `LlamaIndex <https://pypi.org/project/llama-index/>`_ package. Please note that this package will **not be installed** when installing **pymupdf4llm**.
 
     For details on the possible arguments, please consult the LlamaIndex documentation [#f1]_.
 

docs/src/changes.rst

@@ -0,0 +1,27 @@
+.. include:: header.rst
+
+
+Change Log
+===========================================================================
+
+Changes in version 0.0.6
+--------------------------
+
+Fixes:
+~~~~~~~
+
+* `55 <https://github.com/pymupdf/RAG/issues/55>`_ "Bug in helpers/multi_column.py - IndexError: list index out of range"
+* `54 <https://github.com/pymupdf/RAG/issues/54>`_ "Mistakes in orchestrating sentences"
+* `52 <https://github.com/pymupdf/RAG/issues/52>`_ "Chunking of text files"
+* Partial fix for `41 <https://github.com/pymupdf/RAG/issues/41>`_ / `40 <https://github.com/pymupdf/RAG/issues/40>`_. Improved page column detection, but still no silver bullet for overly complex page layouts.
+
+Improvements:
+~~~~~~~~~~~~~~~~
+
+* New parameter `dpi` to specify the resolution of images.
+* New parameters `page_width` / `page_height` for easily processing reflowable documents (Text, Office, e-books).
+* New parameter `graphics_limit` to avoid spending runtimes for value-less content.
+* New parameter `table_strategy` to directly control the table detection strategy.
+
+.. include:: footer.rst
+

docs/src/index.rst

@@ -154,6 +154,13 @@ API
 
 See :doc:`api`.
 
+
+Change Log
+------------
+
+See :doc:`changes`.
+
+
 Further Resources
 -------------------
 

pymupdf4llm/pymupdf4llm/__init__.py

@@ -1,6 +1,6 @@
-from .helpers.pymupdf_rag import to_markdown, IdentifyHeaders
+from .helpers.pymupdf_rag import IdentifyHeaders, to_markdown
 
-__version__ = "0.0.5"
+__version__ = "0.0.6"
 version = __version__
 version_tuple = tuple(map(int, version.split(".")))
 

pymupdf4llm/pymupdf4llm/helpers/get_text_lines.py

@@ -64,15 +64,20 @@ def get_raw_lines(textpage, clip=None, tolerance=3):
     ]
     spans = []  # all spans in TextPage here
     for bno, b in enumerate(blocks):
-        for lno, l in enumerate(b["lines"]):
-            for s in l["spans"]:
+        for lno, line in enumerate(b["lines"]):
+            lbbox = fitz.Rect(line["bbox"])
+            for sno, s in enumerate(line["spans"]):
                 sbbox = fitz.Rect(s["bbox"])  # turn to a Rect
-                if (
-                    abs(sbbox & clip) < abs(sbbox) * 0.8
-                ):  # must be inside parameter rectangle
+                mpoint = (sbbox.tl + sbbox.br) / 2  # middle point
+                if mpoint not in clip:
                     continue
                 if is_white(s["text"]):  # ignore white text
                     continue
+                if s["flags"] & 1 == 1:  # if a superscript, modify
+                    i = 1 if sno == 0 else sno - 1
+                    neighbor = line["spans"][i]
+                    sbbox.y1 = neighbor["bbox"][3]
+                    s["text"] = f"[{s['text']}]"
                 s["bbox"] = sbbox  # update with the Rect version
                 # include line identifier to facilitate separator insertion
                 s["line"] = lno
@@ -82,7 +87,9 @@ def get_raw_lines(textpage, clip=None, tolerance=3):
     if not spans:  # we may have no text at all
         return []
 
-    spans.sort(key=lambda s: s["bbox"].y1)  # sort spans by assending bottom coord
+    spans.sort(
+        key=lambda s: s["bbox"].y1
+    )  # sort spans by assending bottom coord
     nlines = []  # final result
     line = [spans[0]]  # collects spans with fitting vertical coordinate
     lrect = spans[0]["bbox"]  # rectangle joined from span rectangles
@@ -91,7 +98,10 @@ def get_raw_lines(textpage, clip=None, tolerance=3):
         sbbox = s["bbox"]
         sbbox0 = line[-1]["bbox"]
         # if any of top or bottom coordinates are close enough, join...
-        if abs(sbbox.y1 - sbbox0.y1) <= y_delta or abs(sbbox.y0 - sbbox0.y0) <= y_delta:
+        if (
+            abs(sbbox.y1 - sbbox0.y1) <= y_delta
+            or abs(sbbox.y0 - sbbox0.y0) <= y_delta
+        ):
             line.append(s)  # append to this line
             lrect |= sbbox  # extend line rectangle
             continue
@@ -112,7 +122,9 @@ def get_raw_lines(textpage, clip=None, tolerance=3):
     return nlines
 
 
-def get_text_lines(page, *, textpage=None, clip=None, sep="\t", tolerance=3, ocr=False):
+def get_text_lines(
+    page, *, textpage=None, clip=None, sep="\t", tolerance=3, ocr=False
+):
     """Extract text by line keeping natural reading sequence.
 
     Notes: