zot.everything doesn't seem to be working

[yusufaliozkan]

Platform: Jupyter notebook
Python version: 3.10.6
Pyzotero version: 1.5.25

Problem Description

• What were you trying to do?
  • I was trying to retrieve all items in a shared group library using zot.everything(zot.top(limit=10)). When I try, it gives Error 500 suggesting the error is related to the Zotero server. It was working fine until this morning. There was an issue with the Zotero server discussed in the forum this morning. I'm not entirely sure if the error I got is related to the server issue but other elements of pyzotero are working fine.

[urschrei]

Hard to say without seeing the error, but it seems likely to be on the server side if that call was working for you previously.

[yusufaliozkan]

Thanks for the swift response!

The error message is as follows: Could the server error be solely responsible for zot.everything producing an error?

HTTPError: 500 Server Error: Internal Server Error for url: https://api.zotero.org/groups/2514686/items/top?limit=10&start=30&locale=en-US

The above exception was the direct cause of the following exception:

HTTPError                                 Traceback (most recent call last)
Cell In [3], [line 6](vscode-notebook-cell:?execution_count=3&line=6)
      [3](vscode-notebook-cell:?execution_count=3&line=3) api_key = '' # api_key is only needed for private groups and libraries
      [4](vscode-notebook-cell:?execution_count=3&line=4) zot = zotero.Zotero(library_id, library_type)
----> [6](vscode-notebook-cell:?execution_count=3&line=6) items = zot.everything(zot.top(limit=10))
      [7](vscode-notebook-cell:?execution_count=3&line=7) # items = zot.items()
      [8](vscode-notebook-cell:?execution_count=3&line=8) data3=[]

File c:\Users\yaozk\AppData\Local\Programs\Python\Python310\lib\site-packages\pyzotero\zotero.py:865, in Zotero.everything(self, query)
    [863](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:863)     items.extend(query)
    [864](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:864)     while self.links.get("next"):
--> [865](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:865)         items.extend(self.follow())
    [866](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:866) except TypeError:
    [867](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:867)     # we have a bibliography object ughh
    [868](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:868)     items = copy.deepcopy(query)

File c:\Users\yaozk\AppData\Local\Programs\Python\Python310\lib\site-packages\pyzotero\zotero.py:154, in retrieve.<locals>.wrapped_f(self, *args, **kwargs)
    [152](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:152) if kwargs:
    [153](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:153)     self.add_parameters(**kwargs)
--> [154](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:154) retrieved = self._retrieve_data(func(self, *args))
    [155](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:155) # we now always have links in the header response
    [156](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:156) self.links = self._extract_links()

File c:\Users\yaozk\AppData\Local\Programs\Python\Python310\lib\site-packages\pyzotero\zotero.py:407, in Zotero._retrieve_data(self, request, params)
    [405](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:405)     self.request.raise_for_status()
    [406](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:406) except requests.exceptions.HTTPError as exc:
--> [407](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:407)     error_handler(self, self.request, exc)
    [408](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:408) backoff = self.request.headers.get("backoff") or self.request.headers.get(
    [409](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:409)     "retry-after"
    [410](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:410) )
    [411](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:411) if backoff:

File c:\Users\yaozk\AppData\Local\Programs\Python\Python310\lib\site-packages\pyzotero\zotero.py:1644, in error_handler(zot, req, exc)
   [1642](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:1642)     raise ze.HTTPError(err_msg(req))
   [1643](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:1643) else:
-> [1644](file:///C:/Users/yaozk/AppData/Local/Programs/Python/Python310/lib/site-packages/pyzotero/zotero.py:1644)     raise ze.HTTPError(err_msg(req)) from exc

HTTPError: 
Code: 500
URL: https://api.zotero.org/groups/2514686/items/top?limit=10&start=30&locale=en-US
Method: GET
Response: An error occurred

[urschrei]

Yep – HTTP errors are caught but surfaced so that you (or I) can see what went wrong. In this case, we really don't have much to go on since internal errors on the server side typically don't convey much extra information. I will note two things, however:

1. Your .everything() call doesn't make much sense; everything exists to automate the pagination that the Zotero API provides – you typically use it in order to retrieve e.g. all top-level items, but you've restricted it to fetch chunks of 10 at a time. Is there a specific reason for that? What happens when you remove the limit=10 argument?
2. What happens if you call zot.top(start=30, limit=10)?

[yusufaliozkan]

1. The original code snippet is without the limit as it is aimed to retrieve all items. I just put the limit to test if it run without an error. When I remove the limit=10 argument, I've got the same error.

HTTPError: 
Code: 500
URL: https://api.zotero.org/groups/2514686/items/top?limit=100&start=800&locale=en-US
Method: GET
Response: An error occurred

2. The zot.top(start=30, limit=10) argument still gives the same error.

[urschrei]

In that case it looks to be wholly on the server side – you may be able to bring this to their attention on the forum.

[simeneide]

@yusufaliozkan Did you figure this out? Trying to familiarize myself with pyzotero and getting the same error

[g-simmons]

I encountered this error using the local API.

Python 3.12.7
pyzotero 1.5.28

With Zotero(...,local=True), zot.endpoint gets set to http://localhost:23119/api (seems reasonable)

This code:

zot = zotero.Zotero(
    library_id=os.getenv("ZOTERO_USER_ID"),
    library_type="user",
    api_key=os.getenv("ZOTERO_API_KEY"),
    local=True,
)
print(zot.everything(zot.items()))

Produces this error:

Code: 404
URL: http://localhost:23119/api/api/users/7083111/items?format=json&limit=100&locale=en-US&start=100&locale=en-US
Method: GET
Response: No endpoint found

I noticed api/api/ in the URL.

This patch to build_url seems to solve it (probably not optimally):

  def build_url(base_url, path, args_dict=None):
      """Build a valid URL so we don't have to worry about string concatenation errors and
      leading / trailing slashes etc.
      Returns a list in the structure of urlparse.ParseResult"""
+     if 'localhost' in base_url and path.startswith('/api'):
+         path = path.replace('/api', '')
      if base_url.endswith("/"):
          base_url = base_url[:-1]
      url_parts = list(urlparse(base_url))
      url_parts[2] += path
      if args_dict:
          url_parts[4] = urlencode(args_dict)
      return urlunparse(url_parts)

The error occurred when I did zot.items() (~1700 items) but not for zot.top() with some small limit. Seems to be inconsistent across documents? Haven't isolated it further than this. I haven't tested with the remote API, only using local so far.

[urschrei]

@g-simmons This is fixed in v1.6.0, but has nothing to do with this issue.