Filterlist updating is slow

[troysjanda]

Prerequisites

• I verified that this is not a filter issue (MUST be reported at filter issue tracker)
• This is not a support issue or a question
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
• The issue is not present after wholly disabling uBlock Origin ("uBO") in the browser
• I checked the documentation to understand that the issue I report is not a normal behavior

I tried to reproduce the issue when...

• uBO is the only extension
• uBO with default lists/settings
• using a new, unmodified browser profile

Description

when in the Ubo settings panel on the filters list tab and selecting purge and update, the process slows greatly after the builtin section. Not really sure when this started but seems dramatically slower.

• uBlock Origin: 1.44.5b0
• Firefox: 105
• filterset (summary):
• network: 184643
• cosmetic: 85489
• scriptlet: 21106
• html: 780
• listset (total-discarded, last-updated):
• removed:

• plowe-0: null
  

• added:

• https://raw.githubusercontent.com/troysjanda/MyBlockLists/master/UBOXPI: 482-1, 9m
  

• https://raw.githubusercontent.com/troysjanda/MyBlockLists/master/yahoocomplete.txt: 512-0, 9m
  

• https://www.i-dont-care-about-cookies.eu/abp/: 25366-4, 9m
  

• adguard-spyware-url: 763-3, 10m
  

• block-lan: 44-0, 10m
  

• curben-phishing: 97633-4, 9m
  

• curben-pup: 191-0, 9m
  

• fanboy-social: 23358-9, 9m
  

• ublock-annoyances: 4845-9, 9m
  

• default:

• user-filters: 1-0, never
  

• ublock-filters: 33562-133, 10m
  

• ublock-badware: 4204-0, 10m
  

• ublock-privacy: 268-0, 10m
  

• ublock-abuse: 76-0, 10m
  

• ublock-unbreak: 1870-3, 10m
  

• ublock-quick-fixes: 245-0, 10m
  

• easylist: 63219-250, 10m
  

• easyprivacy: 30615-1195, 9m
  

• urlhaus-1: 6456-0, 9m
  

• filterset (user): [array of 1 redacted]
• trustedset:
• added: [array of 40 redacted]
• hostRuleset:
• added: [array of 3 redacted]
• modifiedUserSettings:
• advancedUserEnabled: true
• showIconBadge: false
• webrtcIPAddressHidden: true
• modifiedHiddenSettings:
• autoCommentFilterTemplate: none
• extensionUpdateForceReload: true
• filterAuthorMode: true
• filterOnHeaders: true
• updateAssetBypassBrowserCache: true
• userResourcesLocation: [redacted]
• supportStats:
• allReadyAfter: 219 ms (selfie)
• maxAssetCacheWait: 237 ms

A specific URL where the issue occurs

inturnal page

Steps to Reproduce

1. open the settings panel
2. select the filter list tab
3. hit purge all caches
4. hit update now
5. after the builtin section seems to dramatically slow

Expected behavior

purge and update

Actual behavior

purge is fine but updating after the builtin section is slow

uBlock Origin version

1.44.5b0

Browser name and version

Firefox 105

Operating System and version

Windows 10 19045.2075 22H2

[gorhill]

EasyList server is having issue, including timing out sometimes. Remote servers issues are out of control of uBO and not a uBO issue.

[mapx-]

stephhawk

Can anyone access https://easylist.to/easylist/easylist.txt ? I cannot update EL and EP in uBO

monzta 

the domain is cached/protected via cloudflare. i'm not sure if that is the issue but when i login 
to the dashboard i see a huge amount of traffic coming from india. and a lot of threats do get blocked 
from there. the system status of cloudflare seems fine too. any advice what to do? 

[stephenhawk8054]

Not sure if the current EasyList's server situation causes this: uBlockOrigin/uAssets#14968

From the troubleshooting information, the user's EL list is trimmed down to 17 filters

Details

default: 
    user-filters: 5-1, never
    easylist: 17-1, 3d.6h.52m
    easyprivacy: 30595-1147, 3d.6h.58m
    plowe-0: 3673-529, 5h.52m
    ublock-abuse: 76-0, 15h.21m
    ublock-badware: 4200-2, 3d.6h.56m
    ublock-filters: 32975-30, 3d.6h.55m
    ublock-privacy: 246-1, 15h.20m
    ublock-quick-fixes: 241-6, 5h.51m
    ublock-unbreak: 1855-29, 3d.6h.54m
    urlhaus-1: 7070-0, 5h.50m

Maybe we can switch to https://secure.fanboy.co.nz/easylist.txt (and https://secure.fanboy.co.nz/easyprivacy.txt) as the first update list temporarily until easylist.to server works normal again?

[dportvine]

From the troubleshooting information, the user's EL list is trimmed down to 17 filters

Maybe it's a bug?

Details

uBlock Origin: 1.44.4
Chromium: 105
filterset (summary): 
  network: 100844
  cosmetic: 126563
  scriptlet: 24039
  html: 0
listset (total-discarded, last updated): 
  removed: 
    plowe-0: null
    urlhaus-1: null
  added: 
    https://easylist-downloads.adblockplus.org/bitblock.txt: 31411-27, 5m
    https://easylist-downloads.adblockplus.org/cntblock.txt: 1510-24, 5m
    fanboy-cookiemonster: 41090-287, 5m
    ublock-annoyances: 4827-52, 5m
  default: 
    user-filters: 14-0, never
    RUS-0: 42158-146, 6m
    easylist: 62950-86, 5m
    easyprivacy: 30630-1570, 5m
    ublock-abuse: 76-1, 5m
    ublock-badware: 4200-103, 5m
    ublock-filters: 32865-171, 5m
    ublock-privacy: 246-8, 5m
    ublock-quick-fixes: 233-9, 5m
    ublock-unbreak: 1859-45, 5m
filterset (user): [array of 13 redacted]
modifiedUserSettings: 
  advancedUserEnabled: true
  contextMenuEnabled: false
  showIconBadge: false
modifiedHiddenSettings: 
  filterAuthorMode: true
  popupFontSize: 13px
  popupPanelHeightMode: 1
supportStats: 
  allReadyAfter: 803 ms (selfie)
  maxAssetCacheWait: 647 ms

[stephenhawk8054]

Maybe it's a bug?

I'm not sure. Looks like not everyone has truncated lists issue. Some others have reported missing filters in EL list (one didn't have reddit.com##.promotedlink and one reported having 26k filters instead of 60k):

https://www.reddit.com/r/uBlockOrigin/comments/xgnupg/is_anyone_else_suddenly_getting_reddit_promoted/iot4b1p/?context

[Felipefpl]

The download of fanboy annoyances' list is slow too.

[uBlock-user]

https://adguard.com/en/blog/easylist-filter-problem-help.html

[MaximilianKohler]

I'm having this problem with the built-in filters too:

I noticed ads in Gmail and clicked to refresh the lists and they spin endlessly. Everything else about my internet and browser seems to be normal.

[gothic-bum]

Are you opening the uBO dashboard in a private/incognito window or tab?

[MaximilianKohler]

Yes. ublock has permission to run in private windows. But I just tried in a non-private window, and the updates worked. Thanks.

For some reason, I'm still seeing ads in Gmail though.

[stephenhawk8054]

Yes. ublock has permission to run in private windows. But I just tried in a non-private window, and the updates worked. Thanks.

There's a technical issue in which it affects how uBO marks list updating as finished in private mode and fixing it would just cause other issues. I don't remember where it was discussed though, cc @gorhill

For some reason, I'm still seeing ads in Gmail though.

uBlockOrigin/uAssets#33639 (comment)

Don't discuss websites' filter lists issues here btw. There's another repo for websites' issues.