src_fetch() phase function to support VCS fetching
Hello,
As Sid Hayn raised today on #gentoo-portage, it would be useful to
finally have portage able to fetch updates from VCS-es independently
of src_unpack(). This could be used, for example, on machines
temporarily connected to the network -- one would then fetch files
while connected to the network, and perform the updates later.
There are a few ways how we could handle that but the cleanest and most
universal one seems to be defining a src_fetch() phase function
in a future EAPI.
In the EAPIs supporting src_fetch(), that phase function would be used
by PM when requesting the files to be fetched. A default_src_fetch()
will be declared as well, providing implementation-defined code
fetching files like they are fetched now. Older EAPIs will simply
always use that default.
The phase function would be disjoint from the normal merge process,
much like pkg_pretend(). In portage, it will be called as 'portage'
user if FEATURES=userfetch is enabled.
VCS eclasses supporting separated fetching would define two phase
functions:
- src_fetch() which would be responsible for fetching updates,
- src_unpack() which would be responsible for checking out the source
to work directory.
The remaining issue is handling dependencies on the tools necessary to
do fetching. For default_src_fetch(), we can assume that the package
manager provides the necessary tools. For custom src_fetch(), we would
need either to:
1) require satisfying whole DEPEND when fetching -- probably pointless,
as it will make --fetchonly almost impossible when doing initial
installs;
2) introduce a new dependency type (please do not get into details how
we do it -- we will discuss that another time, at the moment please
just keep it as 'new dependency type') -- and we probably end up
having a switch for --fetchonly without installing deps (thus
omitting packages where they are not satisfied), and with deps;
3) [ugly!] assume that src_fetch() should check for its deps and fail
if they are not satisfied. If that's mostly for live ebuilds, it may
be acceptable. Then the package manager will just have one 'fetch
failed' on --fetchonly (or early pre-fetch), and it will have to
invoke src_fetch() after satisfying the deps, before src_unpack().
What do you think? What are your ideas, suggestions?
--
Best regards,
Michał Górny
09-04-2012, 05:02 PM
"Rick "Zero_Chaos" Farina"
src_fetch() phase function to support VCS fetching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/04/2012 12:43 PM, Michał Górny wrote:
> Hello,
>
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
If you insist on using real names mine is Rick ;-)
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
>
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
>
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
>
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
>
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
> to work directory.
>
> The remaining issue is handling dependencies on the tools necessary to
> do fetching. For default_src_fetch(), we can assume that the package
> manager provides the necessary tools. For custom src_fetch(), we would
> need either to:
>
> 1) require satisfying whole DEPEND when fetching -- probably pointless,
> as it will make --fetchonly almost impossible when doing initial
> installs;
>
> 2) introduce a new dependency type (please do not get into details how
> we do it -- we will discuss that another time, at the moment please
> just keep it as 'new dependency type') -- and we probably end up
> having a switch for --fetchonly without installing deps (thus
> omitting packages where they are not satisfied), and with deps;
>
> 3) [ugly!] assume that src_fetch() should check for its deps and fail
> if they are not satisfied. If that's mostly for live ebuilds, it may
> be acceptable. Then the package manager will just have one 'fetch
> failed' on --fetchonly (or early pre-fetch), and it will have to
> invoke src_fetch() after satisfying the deps, before src_unpack().
>
> What do you think? What are your ideas, suggestions?
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
src_fetch() phase function to support VCS fetching
On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Hello,
>
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
>
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
>
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
>
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
>
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
> to work directory.
The 'checking out' language for src_unpack() sounds like it assumes a
DVCS such as mercurial or git. What about cvs or svn, where fetching
is also checking out? (This is probably a trivial thing to clear up,
though.)
Also, where would the local copy go? distfiles? It's common for
distfiles to be stored on, e.g. an NFS mount, so you may need to be
careful not to place repositories there which have filesystem
semantics that are disagreeable to NFS. (The only example I know of
off the top of my head is svn, where the documentation warns against
using the dbd backend on top of NFS.)
Other common remote mounts (such as cifs) may have restrictions that
could force munging of filenames, too, and VCS infrastructures (or
even unpacked checkouts with strange filenames) placed on those
filesystems may have unanticipated results.
It may be helpful to have some kind of adapter mount in place, or even
generate a tarball of the local copy and store that instead. (That'd
be problematic if multiple boxes were modifying the local copy on the
same share, but that's obviously problematic anyway.)
--
:wq
09-04-2012, 05:05 PM
"Rick "Zero_Chaos" Farina"
src_fetch() phase function to support VCS fetching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/04/2012 12:43 PM, Michał Górny wrote:
> Hello,
>
> As Sid Hayn raised today on #gentoo-portage, it would be useful to
> finally have portage able to fetch updates from VCS-es independently
> of src_unpack(). This could be used, for example, on machines
> temporarily connected to the network -- one would then fetch files
> while connected to the network, and perform the updates later.
>
> There are a few ways how we could handle that but the cleanest and most
> universal one seems to be defining a src_fetch() phase function
> in a future EAPI.
>
> In the EAPIs supporting src_fetch(), that phase function would be used
> by PM when requesting the files to be fetched. A default_src_fetch()
> will be declared as well, providing implementation-defined code
> fetching files like they are fetched now. Older EAPIs will simply
> always use that default.
>
> The phase function would be disjoint from the normal merge process,
> much like pkg_pretend(). In portage, it will be called as 'portage'
> user if FEATURES=userfetch is enabled.
>
> VCS eclasses supporting separated fetching would define two phase
> functions:
> - src_fetch() which would be responsible for fetching updates,
> - src_unpack() which would be responsible for checking out the source
> to work directory.
>
> The remaining issue is handling dependencies on the tools necessary to
> do fetching. For default_src_fetch(), we can assume that the package
> manager provides the necessary tools. For custom src_fetch(), we would
> need either to:
>
> 1) require satisfying whole DEPEND when fetching -- probably pointless,
> as it will make --fetchonly almost impossible when doing initial
> installs;
>
> 2) introduce a new dependency type (please do not get into details how
> we do it -- we will discuss that another time, at the moment please
> just keep it as 'new dependency type') -- and we probably end up
> having a switch for --fetchonly without installing deps (thus
> omitting packages where they are not satisfied), and with deps;
>
> 3) [ugly!] assume that src_fetch() should check for its deps and fail
> if they are not satisfied. If that's mostly for live ebuilds, it may
> be acceptable. Then the package manager will just have one 'fetch
> failed' on --fetchonly (or early pre-fetch), and it will have to
> invoke src_fetch() after satisfying the deps, before src_unpack().
I believe the easiest (and honestly most sane) method is to simply have
src_fetch in the live classes check for needed deps and die (with a
"please emerge blah") if deps are not found. Adding something like
FDEPEND just seems to be getting way too crazy on the dependency tree
AND would require things to build during fetch-only which doesn't make
sense.
Thanks,
Zero
>
> What do you think? What are your ideas, suggestions?
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
src_fetch() phase function to support VCS fetching
On Tue, 4 Sep 2012 13:02:36 -0400
Michael Mol <mikemol@gmail.com> wrote:
> On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org>
> wrote:
> > Hello,
> >
> > As Sid Hayn raised today on #gentoo-portage, it would be useful to
> > finally have portage able to fetch updates from VCS-es independently
> > of src_unpack(). This could be used, for example, on machines
> > temporarily connected to the network -- one would then fetch files
> > while connected to the network, and perform the updates later.
> >
> > There are a few ways how we could handle that but the cleanest and
> > most universal one seems to be defining a src_fetch() phase function
> > in a future EAPI.
> >
> > In the EAPIs supporting src_fetch(), that phase function would be
> > used by PM when requesting the files to be fetched. A
> > default_src_fetch() will be declared as well, providing
> > implementation-defined code fetching files like they are fetched
> > now. Older EAPIs will simply always use that default.
> >
> > The phase function would be disjoint from the normal merge process,
> > much like pkg_pretend(). In portage, it will be called as 'portage'
> > user if FEATURES=userfetch is enabled.
> >
> > VCS eclasses supporting separated fetching would define two phase
> > functions:
> > - src_fetch() which would be responsible for fetching updates,
> > - src_unpack() which would be responsible for checking out the
> > source to work directory.
>
> The 'checking out' language for src_unpack() sounds like it assumes a
> DVCS such as mercurial or git. What about cvs or svn, where fetching
> is also checking out? (This is probably a trivial thing to clear up,
> though.)
They either stay with src_unpack() or do 'cvs up' in src_fetch()
and just copy files over in src_unpack(). Anyway, that's what they do
now -- update the copy in distfiles/cvs-src and then copy it.
> Also, where would the local copy go? distfiles? It's common for
> distfiles to be stored on, e.g. an NFS mount, so you may need to be
> careful not to place repositories there which have filesystem
> semantics that are disagreeable to NFS. (The only example I know of
> off the top of my head is svn, where the documentation warns against
> using the dbd backend on top of NFS.)
The actual code will be eclass-dependent, so it will go whatever it
goes now. No magic, no exherbo. Just plain phase function.
--
Best regards,
Michał Górny
09-04-2012, 05:24 PM
"Rick "Zero_Chaos" Farina"
src_fetch() phase function to support VCS fetching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/04/2012 01:02 PM, Michael Mol wrote:
> On Tue, Sep 4, 2012 at 12:43 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> Hello,
>>
>> As Sid Hayn raised today on #gentoo-portage, it would be useful to
>> finally have portage able to fetch updates from VCS-es independently
>> of src_unpack(). This could be used, for example, on machines
>> temporarily connected to the network -- one would then fetch files
>> while connected to the network, and perform the updates later.
>>
>> There are a few ways how we could handle that but the cleanest and most
>> universal one seems to be defining a src_fetch() phase function
>> in a future EAPI.
>>
>> In the EAPIs supporting src_fetch(), that phase function would be used
>> by PM when requesting the files to be fetched. A default_src_fetch()
>> will be declared as well, providing implementation-defined code
>> fetching files like they are fetched now. Older EAPIs will simply
>> always use that default.
>>
>> The phase function would be disjoint from the normal merge process,
>> much like pkg_pretend(). In portage, it will be called as 'portage'
>> user if FEATURES=userfetch is enabled.
>>
>> VCS eclasses supporting separated fetching would define two phase
>> functions:
>> - src_fetch() which would be responsible for fetching updates,
>> - src_unpack() which would be responsible for checking out the source
>> to work directory.
>
> The 'checking out' language for src_unpack() sounds like it assumes a
> DVCS such as mercurial or git. What about cvs or svn, where fetching
> is also checking out? (This is probably a trivial thing to clear up,
> though.)
>
> Also, where would the local copy go? distfiles? It's common for
> distfiles to be stored on, e.g. an NFS mount, so you may need to be
> careful not to place repositories there which have filesystem
> semantics that are disagreeable to NFS. (The only example I know of
> off the top of my head is svn, where the documentation warns against
> using the dbd backend on top of NFS.)
>
> Other common remote mounts (such as cifs) may have restrictions that
> could force munging of filenames, too, and VCS infrastructures (or
> even unpacked checkouts with strange filenames) placed on those
> filesystems may have unanticipated results.
>
> It may be helpful to have some kind of adapter mount in place, or even
> generate a tarball of the local copy and store that instead. (That'd
> be problematic if multiple boxes were modifying the local copy on the
> same share, but that's obviously problematic anyway.)
>
All the live eclasses already drop a checkout (or whatever term you
like) of the repo in /usr/portage/distfiles. What we are talking about
here is separating the "download/checkout into /usr/portage/distfiles"
from the "copy from /usr/portage/distfiles to ${S}". Making this two
separate phases would allow a reasonably sane "emerge -f blah" support
to fetch the live sources before build.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
src_fetch() phase function to support VCS fetching
On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
> I believe the easiest (and honestly most sane) method is to simply have
> src_fetch in the live classes check for needed deps and die (with a
> "please emerge blah") if deps are not found. Adding something like
> FDEPEND just seems to be getting way too crazy on the dependency tree
> AND would require things to build during fetch-only which doesn't make
> sense.
I think it's nicer to have FDEPEND because it makes the deps more
complete, so the package manager can bail out when necessary, without
even executing src_fetch. In the case of --fetchonly the package manager
could simply bail out if the deps are not installed (like how it bails
out for --buildpkgonly when the deps aren't installed).
--
Thanks,
Zac
09-04-2012, 07:56 PM
Ian Stakenvicius
src_fetch() phase function to support VCS fetching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 04/09/12 01:32 PM, Zac Medico wrote:
> On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
>> I believe the easiest (and honestly most sane) method is to
>> simply have src_fetch in the live classes check for needed deps
>> and die (with a "please emerge blah") if deps are not found.
>> Adding something like FDEPEND just seems to be getting way too
>> crazy on the dependency tree AND would require things to build
>> during fetch-only which doesn't make sense.
>
> I think it's nicer to have FDEPEND because it makes the deps more
> complete, so the package manager can bail out when necessary,
> without even executing src_fetch. In the case of --fetchonly the
> package manager could simply bail out if the deps are not installed
> (like how it bails out for --buildpkgonly when the deps aren't
> installed).
Just looking into the future here; would things like archivers or
other helpers used by src_unpack move to FDEPEND as well? or would
this be limited solely to tools that data transfer?
src_fetch() phase function to support VCS fetching
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/04/2012 03:56 PM, Ian Stakenvicius wrote:
> On 04/09/12 01:32 PM, Zac Medico wrote:
>> On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
>>> I believe the easiest (and honestly most sane) method is to
>>> simply have src_fetch in the live classes check for needed deps
>>> and die (with a "please emerge blah") if deps are not found.
>>> Adding something like FDEPEND just seems to be getting way too
>>> crazy on the dependency tree AND would require things to build
>>> during fetch-only which doesn't make sense.
>
>> I think it's nicer to have FDEPEND because it makes the deps more
>> complete, so the package manager can bail out when necessary,
>> without even executing src_fetch. In the case of --fetchonly the
>> package manager could simply bail out if the deps are not installed
>> (like how it bails out for --buildpkgonly when the deps aren't
>> installed).
>
> Just looking into the future here; would things like archivers or
> other helpers used by src_unpack move to FDEPEND as well? or would
> this be limited solely to tools that data transfer?
We are talking about things required for src_fetch (the download) so no,
things required for src_unpack have no real place in this as far as I am
concerned.
>
>
>
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
src_fetch() phase function to support VCS fetching
On Tue, 04 Sep 2012 15:56:54 -0400
Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 04/09/12 01:32 PM, Zac Medico wrote:
> > On 09/04/2012 10:05 AM, Rick "Zero_Chaos" Farina wrote:
> >> I believe the easiest (and honestly most sane) method is to
> >> simply have src_fetch in the live classes check for needed deps
> >> and die (with a "please emerge blah") if deps are not found.
> >> Adding something like FDEPEND just seems to be getting way too
> >> crazy on the dependency tree AND would require things to build
> >> during fetch-only which doesn't make sense.
> >
> > I think it's nicer to have FDEPEND because it makes the deps more
> > complete, so the package manager can bail out when necessary,
> > without even executing src_fetch. In the case of --fetchonly the
> > package manager could simply bail out if the deps are not installed
> > (like how it bails out for --buildpkgonly when the deps aren't
> > installed).
>
> Just looking into the future here; would things like archivers or
> other helpers used by src_unpack move to FDEPEND as well? or would
> this be limited solely to tools that data transfer?
src_fetch() only. Archivers belong to src_unpack(), which belongs to
the main phase function cycle -> DEPEND.
src_fetch() phase function to support VCS fetching
