Mike Fährmann
7c1cb923a4
[myportfolio] replace unit test
...
the old gallery got removed
6 years ago
Mike Fährmann
fffbfd3dce
[imgspice] fix extraction
6 years ago
Mike Fährmann
4ca4631bad
simplify auto-disabling certificate verification
...
if no certificate bundle is found
6 years ago
Mike Fährmann
09d872a2b1
generalize extractor creation code
6 years ago
Mike Fährmann
8dc6be246b
[shopify] add custom retry logic for 430 status codes ( #175 )
6 years ago
Mike Fährmann
0887fb61f4
[komikcast] update test results
6 years ago
Mike Fährmann
a881537b91
more util.py tests
6 years ago
Mike Fährmann
976ccb267f
[myportfolio] combine gallery and user extractors
...
An URL alone isn't good enough to distinguish between a gallery or a
gallery-listing, so the new extractor decides what to do based on the
page's content.
6 years ago
Mike Fährmann
efd104e45e
[instagram] reject more non-user URLs ( #180 )
6 years ago
HRXN
56e0e92e0d
[shopify] cosmetic changes in shopify.py ( #181 )
...
Glanced over the commits, randomly spotted some minor things.
6 years ago
Mike Fährmann
23baecb29e
fix 'CONVERSIONS' variable name
6 years ago
Mike Fährmann
9c0e2f294b
[shopify] add generic collection and product extractors ( #175 )
...
with fashionnova.com as a default domain
6 years ago
Mike Fährmann
105097ddcf
add 'S' conversion options for format string fields
...
Same as 's' (convert to string), but has a better, human-readable
conversion for lists.
6 years ago
Mike Fährmann
1578013efc
remove unused default config path
6 years ago
Mike Fährmann
26c4365baa
adjust metadata types for GalleryExtractors
6 years ago
Mike Fährmann
13e0f2a78f
[deviantart] add 'scraps' extractor ( closes #168 )
6 years ago
Mike Fährmann
3ea11f5d5e
[nhentai] rewrite
...
- use GalleryExtractor as base class
- extract a lot more metadata (artist, tags, etc.)
6 years ago
Mike Fährmann
176b7253a1
update function signature for config.load()
6 years ago
Mike Fährmann
3595cd582f
use GalleryExtractor as common base class
6 years ago
林博仁(Buo-ren Lin)
77551bf01b
Implement snap packaging, snaps are universal linux packages ( #170 )
...
This patch implements the necessary details to package gallery-dl as a snap that can be run on a broad range of supporting GNU/Linux distributions[1].
[1] https://snapcraft.io/
Signed-off-by: 林博仁(Buo-ren Lin) <Buo.Ren.Lin@gmail.com>
6 years ago
Mike Fährmann
a138d5873d
[hentaifoundry] improve/fix extraction
...
- Sometimes an ad interfered when trying to get a download URL
- Resolving "www.hentai-foundry.com" yields an invalid(?) IPv6 address
(2607:5300:60:ca9e:feed:dead:beef:1) and urllib3 only tries to connect
to the IPv4 variant after a rather long wait time
6 years ago
Mike Fährmann
280531c8ff
[pururin] add gallery extractor ( closes #174 )
6 years ago
Mike Fährmann
3159dd79d5
[seiga] use HTTPS
6 years ago
Mike Fährmann
ea6fbfdd39
simplify build_supportedsites.py
6 years ago
Mike Fährmann
ee2ad76620
update and improve create_test_data.py script
6 years ago
Mike Fährmann
f6734142ee
[komikcast] remove 'width' and 'height' info
6 years ago
Mike Fährmann
d0059cab79
[tumblr] check for null URLs ( closes #165 )
6 years ago
Mike Fährmann
e687a6095e
[luscious] raise exception if album is not available
6 years ago
Mike Fährmann
22d3a2fcc8
[artstation] add extractor for artwork listings ( #80 )
...
like https://www.artstation.com/artwork?sorting=latest
or https://www.artstation.com/artwork?sorting=picks
6 years ago
Mike Fährmann
937a802b49
[dynastyscans] add extractors for images and image searches
...
(closes #163 )
6 years ago
Mike Fährmann
b09a8184ca
move TestJob into test module; test _extractor values
6 years ago
Mike Fährmann
19860655a3
[weibo] add 'user' and 'status' extractors
6 years ago
Mike Fährmann
f8782c05f2
[paheal] rename "tags" to "search_tags"
...
to better match field names of other booru extractors
6 years ago
Mike Fährmann
c7b8421333
[deviantart] don't match 'www' as a potential username
6 years ago
Mike Fährmann
5530871b5a
change results of text.nameext_from_url()
...
Instead of getting a complete 'filename' from an URL and splitting that
into 'name' and 'extension', the new approach gets rid of the complete
version and renames 'name' to 'filename'. (Using anything other than
{extension} for a filename extension doesn't really work anyway)
Example: "https://example.org/path/filename.ext "
before:
- filename : filename.ext
- name : filename
- extension: ext
now:
- filename : filename
- extension: ext
6 years ago
Mike Fährmann
148b8f15d0
update tests for util.py
6 years ago
Mike Fährmann
ae353ed3b0
provide "extractor" and "job" keys for logging output
...
This allows for stuff like "{extractor.url}" and "{extractor.category}"
in logging format strings.
Accessing 'extractor' and 'job' in any way will return "None" if those
fields aren't defined, i.e. in general logging messages.
6 years ago
Mike Fährmann
32edf4fc7b
add '_extractor' info to manga extractor results
6 years ago
Mike Fährmann
89ee8cd7e4
filter "private" kwdict entries
6 years ago
Mike Fährmann
61741d7333
provide type information for Queue messages
...
Child extractors are now directly constructed with Extractor.from_url()
if the extractor class is known beforehand, instead of using
extractor.find() and searching through all possible extractor classes.
6 years ago
Mike Fährmann
2e516a1e3e
store the full original URL in Extractor.url
6 years ago
Mike Fährmann
580baef72c
change Chapter and MangaExtractor classes
...
- unify and simplify constructors
- rename get_metadata and get_images to just metadata() and images()
- rename self.url to chapter_url and manga_url
6 years ago
Mike Fährmann
4b1880fa5e
propagate 'match' to base extractor constructor
6 years ago
Mike Fährmann
ade86da7a1
[tsumino] replace test
6 years ago
Mike Fährmann
1f3422c28b
[mangahere] fix extraction
6 years ago
Mike Fährmann
84ae72b8d8
[ngomik] fix extraction
6 years ago
Mike Fährmann
02d733d219
[simplyhentai] fix and improve tag extraction
...
The "tags" field is now a list instead of a string.
In format strings, use "{tags:J, }" to Join them.
6 years ago
Mike Fährmann
3a0b4af744
[seiga] recognize /thumb/ URLs
...
https://lohas.nicoseiga.jp/thumb/5977527i
6 years ago
Mike Fährmann
8fc6fbfa34
[artstation] recognize shortened project URLs
...
https://artstn.co/p/ <project-id>
6 years ago
Mike Fährmann
9a9cd32461
implement alternative constructor for extractors
6 years ago